[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:04 dist-test-slave-qxh0 krb5kdc[5078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:04 dist-test-slave-qxh0 krb5kdc[5078](info): set up 2 sockets
Sep 25 14:58:04 dist-test-slave-qxh0 krb5kdc[5078](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 14:58:05.611375 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:05 dist-test-slave-qxh0 krb5kdc[5078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:05.781024 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:45883
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:45883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:06.112550 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:06.112848 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:06.113008 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:06.120956 5091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:06.121063 5091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:06.121161 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:06.121266 5091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:06.121347 5091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:06.131266 5091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:45883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:06.132643 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:06.133994 5091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:06.146417 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:06.146441 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:06.146656 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:06.147269 5091 server_base.cc:1034] running on GCE node
I20260925 14:58:06.148067 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:06.149566 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:06.150753 5091 hybrid_clock.cc:648] HybridClock initialized: now 1790348286150712 us; error 30 us; skew 500 ppm
I20260925 14:58:06.151091 5091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:06 dist-test-slave-qxh0 krb5kdc[5078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:06.154138 5091 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:06.155680 5091 webserver.cc:460] Webserver started at http://127.4.243.254:39527/ using document root <none> and password file <none>
I20260925 14:58:06.156354 5091 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:06.156502 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:06.156772 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:06.160044 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "2d76343f34704744b1226f11e5ae4790"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-qxh0"
server_key: "32b2f69e2f29448a7cba39fe01331a33"
server_key_iv: "d8165509d419a5f2c90240507f9405a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.160758 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "2d76343f34704744b1226f11e5ae4790"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-qxh0"
server_key: "32b2f69e2f29448a7cba39fe01331a33"
server_key_iv: "d8165509d419a5f2c90240507f9405a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.165767 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 14:58:06.169456 5106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:06.170894 5091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 14:58:06.171072 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "2d76343f34704744b1226f11e5ae4790"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-qxh0"
server_key: "32b2f69e2f29448a7cba39fe01331a33"
server_key_iv: "d8165509d419a5f2c90240507f9405a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.171242 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:06.203423 5091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:06.208392 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:06.208698 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:06.227873 5091 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:45883
I20260925 14:58:06.227881 5157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:45883 every 8 connection(s)
I20260925 14:58:06.229239 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:06.233426 5158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:06.238521 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5091
I20260925 14:58:06.238863 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:06.240662 5071 external_mini_cluster.cc:1428] Setting key 1898dcb405036ea0569013d42b193019
I20260925 14:58:06.243511 5158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790: Bootstrap starting.
I20260925 14:58:06.247952 5158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:06.248951 5158 log.cc:824] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:06.251814 5158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790: No bootstrap required, opened a new log
I20260925 14:58:06.260655 5158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d76343f34704744b1226f11e5ae4790" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45883 } }
I20260925 14:58:06.261053 5158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:06.261183 5158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d76343f34704744b1226f11e5ae4790, State: Initialized, Role: FOLLOWER
I20260925 14:58:06.261750 5158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d76343f34704744b1226f11e5ae4790" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45883 } }
I20260925 14:58:06.261983 5158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:06.262169 5158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:06.262460 5158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:06.264659 5158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d76343f34704744b1226f11e5ae4790" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45883 } }
I20260925 14:58:06.265209 5158 leader_election.cc:302] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d76343f34704744b1226f11e5ae4790; no voters:
I20260925 14:58:06.265869 5158 leader_election.cc:288] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:06.265978 5163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:06.267791 5163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [term 1 LEADER]: Becoming Leader. State: Replica: 2d76343f34704744b1226f11e5ae4790, State: Running, Role: LEADER
I20260925 14:58:06.268329 5163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d76343f34704744b1226f11e5ae4790" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45883 } }
I20260925 14:58:06.268471 5158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 14:58:06 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:06.273427 5164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d76343f34704744b1226f11e5ae4790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d76343f34704744b1226f11e5ae4790" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45883 } } }
I20260925 14:58:06.273921 5164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:06.274524 5165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d76343f34704744b1226f11e5ae4790. Latest consensus state: current_term: 1 leader_uuid: "2d76343f34704744b1226f11e5ae4790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d76343f34704744b1226f11e5ae4790" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45883 } } }
I20260925 14:58:06.274865 5165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:06.277905 5171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:06.283303 5171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:06.285218 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:06.244523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.4.243.254:45883)
0925 14:58:06.245957 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:06.245976 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:06.247058 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 14:58:06.256817 (+ 9759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:06.256839 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:06.257235 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:06.257680 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:06.260542 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.262154 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.263901 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.264553 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.264772 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.271941 (+ 7169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:06.271954 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:06.271983 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:06.272052 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:06.276848 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.277841 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.277853 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.277862 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.278124 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.278760 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.278770 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.278776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.279546 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:06.279931 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:06.282894 (+ 2963us) server_negotiation.cc:299] Negotiation successful
0925 14:58:06.283434 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":409,"threads_started":1}
I20260925 14:58:06.295871 5171 catalog_manager.cc:1347] Generated new cluster ID: cc91c1e8a0bf424b8d044603d4082141
I20260925 14:58:06.296015 5171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:06.313808 5171 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:06.315202 5171 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:06.324206 5171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d76343f34704744b1226f11e5ae4790: Generated new TSK 0
I20260925 14:58:06.325029 5171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:06.372313 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45883
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:06.740211 5185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:06.740491 5185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:06.740585 5185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:06.748404 5185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:06.748520 5185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:06.748696 5185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:06.759047 5185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:06.760445 5185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:06.761811 5185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:06.775964 5190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:06.776000 5193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:06.776043 5191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:06.777092 5185 server_base.cc:1034] running on GCE node
I20260925 14:58:06.777828 5185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:06.778914 5185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:06.780144 5185 hybrid_clock.cc:648] HybridClock initialized: now 1790348286780099 us; error 35 us; skew 500 ppm
I20260925 14:58:06.780567 5185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:06 dist-test-slave-qxh0 krb5kdc[5078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:06.783650 5185 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:06.785070 5185 webserver.cc:460] Webserver started at http://127.4.243.193:36755/ using document root <none> and password file <none>
I20260925 14:58:06.785724 5185 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:06.785871 5185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:06.786190 5185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:06.789119 5185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "e91d8e1ede294d62aa6172313537bd06"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-qxh0"
server_key: "6f16e3181be37aad5735e448c8938a0a"
server_key_iv: "6a97cfba85e917ead52067ff2e34ddf0"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.790324 5185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "e91d8e1ede294d62aa6172313537bd06"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-qxh0"
server_key: "6f16e3181be37aad5735e448c8938a0a"
server_key_iv: "6a97cfba85e917ead52067ff2e34ddf0"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.795584 5185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 14:58:06.798946 5200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:06.800273 5185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:58:06.800450 5185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "e91d8e1ede294d62aa6172313537bd06"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-qxh0"
server_key: "6f16e3181be37aad5735e448c8938a0a"
server_key_iv: "6a97cfba85e917ead52067ff2e34ddf0"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.800658 5185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:06.816977 5185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:06.821434 5185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:06.821760 5185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:06.822863 5185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:06.824762 5185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:06.824865 5185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:06.825017 5185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:06.825089 5185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:06.860492 5185 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:33323
I20260925 14:58:06.860507 5312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:33323 every 8 connection(s)
I20260925 14:58:06.861855 5185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:06.862480 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5185
I20260925 14:58:06.862769 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:06.863763 5071 external_mini_cluster.cc:1428] Setting key 453cc93231c950877d1fce62e2b9a020
Sep 25 14:58:06 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 14:58:06.883769 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:06.864840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:42511 (local address 127.4.243.254:45883)
0925 14:58:06.865329 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:06.865334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:06.866562 (+ 1228us) server_negotiation.cc:407] Connection header received
0925 14:58:06.868451 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:06.868459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:06.868546 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:06.868684 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:06.870389 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.871652 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.873217 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.873972 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.874126 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.878101 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:06.878106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:06.878110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:06.878143 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:06.880744 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.881441 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.881447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.881451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.881566 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.881978 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.881983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.881986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.882243 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:06.882401 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:06.883196 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 14:58:06.883402 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
I20260925 14:58:06.884742 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:06.865516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45883 (local address 127.4.243.193:42511)
0925 14:58:06.866384 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:06.866431 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:06.868128 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:06.868900 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:06.868908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:06.869515 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:06.870217 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:06.870235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.871819 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.871824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:06.873070 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:06.873082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.874194 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.874199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:06.874458 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.875308 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:06.875332 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:06.877874 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:06.880890 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:06.880899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:06.880918 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:06.881323 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:06.881675 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:06.881680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:06.881684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:06.881864 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:06.882376 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:06.882384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:06.882792 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 14:58:06.883107 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 14:58:06.883637 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":358,"threads_started":1}
I20260925 14:58:06.886379 5313 heartbeater.cc:347] Connected to a master server at 127.4.243.254:45883
I20260925 14:58:06.886770 5313 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:06.887494 5313 heartbeater.cc:511] Master 127.4.243.254:45883 requested a full tablet report, sending...
I20260925 14:58:06.889889 5123 ts_manager.cc:194] Registered new tserver with Master: e91d8e1ede294d62aa6172313537bd06 (127.4.243.193:33323)
I20260925 14:58:06.891454 5123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:42511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:06.905557 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45883
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:07.236605 5319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:07.236867 5319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:07.237021 5319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:07.244678 5319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:07.244791 5319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:07.244951 5319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:07.255115 5319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:07.256438 5319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:07.257761 5319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:07.271343 5324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.271446 5327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.271368 5325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.272450 5319 server_base.cc:1034] running on GCE node
I20260925 14:58:07.273048 5319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:07.273926 5319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:07.275182 5319 hybrid_clock.cc:648] HybridClock initialized: now 1790348287275096 us; error 34 us; skew 500 ppm
I20260925 14:58:07.275449 5319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:07 dist-test-slave-qxh0 krb5kdc[5078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:07.278486 5319 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:07.280040 5319 webserver.cc:460] Webserver started at http://127.4.243.194:36017/ using document root <none> and password file <none>
I20260925 14:58:07.280697 5319 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:07.280821 5319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:07.281551 5319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:07.284359 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "78354a695ee846e0be743e954041edbf"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-qxh0"
server_key: "0d5d45ed0b791c918181d4e368222073"
server_key_iv: "e753e5905d910e40003a2823b731636b"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.285099 5319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "78354a695ee846e0be743e954041edbf"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-qxh0"
server_key: "0d5d45ed0b791c918181d4e368222073"
server_key_iv: "e753e5905d910e40003a2823b731636b"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.289973 5319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 14:58:07.293479 5334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.294906 5319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:07.295074 5319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "78354a695ee846e0be743e954041edbf"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-qxh0"
server_key: "0d5d45ed0b791c918181d4e368222073"
server_key_iv: "e753e5905d910e40003a2823b731636b"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.295284 5319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:07.320988 5319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:07.325290 5319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:07.325594 5319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:07.326581 5319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:07.328020 5319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:07.328092 5319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.328277 5319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:07.328327 5319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.364259 5319 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:35931
I20260925 14:58:07.364346 5446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:35931 every 8 connection(s)
I20260925 14:58:07.365602 5319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:07.372913 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5319
I20260925 14:58:07.373163 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:07.373580 5071 external_mini_cluster.cc:1428] Setting key 27776fc7215336bbababfec942080a59
Sep 25 14:58:07 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:07.388975 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.368447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:42681 (local address 127.4.243.254:45883)
0925 14:58:07.368623 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.368629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.369923 (+ 1294us) server_negotiation.cc:407] Connection header received
0925 14:58:07.372422 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.372428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.372525 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.372652 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.374467 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.375384 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.377117 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.377961 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.378104 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.382710 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.382714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.382717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.382749 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.385390 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.386387 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.386392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.386396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.386525 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.387063 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.387067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.387069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.387381 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.387508 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.388540 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.388725 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:58:07.390702 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.368939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45883 (local address 127.4.243.194:42681)
0925 14:58:07.369756 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.369797 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.372103 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.372886 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.372896 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.373606 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:07.374299 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.374318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.375567 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.375573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.376956 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.376973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.378163 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.378169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.378382 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.379163 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:07.379196 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:07.382414 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:07.385536 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.385548 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.385569 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.386239 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.386662 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.386670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.386674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.386927 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.387527 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:07.387538 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:07.388059 (+ 521us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.388452 (+ 393us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.389064 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:07.392390 5447 heartbeater.cc:347] Connected to a master server at 127.4.243.254:45883
I20260925 14:58:07.392720 5447 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:07.393509 5447 heartbeater.cc:511] Master 127.4.243.254:45883 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 14:58:07.395265 5123 ts_manager.cc:194] Registered new tserver with Master: 78354a695ee846e0be743e954041edbf (127.4.243.194:35931)
I20260925 14:58:07.396080 5123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:42681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:07.416093 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45883
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:07.756284 5452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:07.756554 5452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:07.756647 5452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:07.764099 5452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:07.764216 5452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:07.764330 5452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:07.773962 5452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:07.775355 5452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:07.776667 5452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:07.790009 5457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.790040 5458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.790077 5460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.790745 5452 server_base.cc:1034] running on GCE node
I20260925 14:58:07.791252 5452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:07.792089 5452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:07.793301 5452 hybrid_clock.cc:648] HybridClock initialized: now 1790348287793240 us; error 53 us; skew 500 ppm
I20260925 14:58:07.793603 5452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:07 dist-test-slave-qxh0 krb5kdc[5078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:07.797042 5452 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:07.798497 5452 webserver.cc:460] Webserver started at http://127.4.243.195:44509/ using document root <none> and password file <none>
I20260925 14:58:07.799173 5452 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:07.799321 5452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:07.799623 5452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:07.802367 5452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "94423c72d1e54700ab73199acbe63dd2"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-qxh0"
server_key: "71a1c1a7af4770e744899aa93906af5d"
server_key_iv: "771bffc1ef239e5b99c1cbe97493ed04"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.803143 5452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "94423c72d1e54700ab73199acbe63dd2"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-qxh0"
server_key: "71a1c1a7af4770e744899aa93906af5d"
server_key_iv: "771bffc1ef239e5b99c1cbe97493ed04"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.808357 5452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:58:07.811815 5467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.813180 5452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:07.813372 5452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "94423c72d1e54700ab73199acbe63dd2"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-qxh0"
server_key: "71a1c1a7af4770e744899aa93906af5d"
server_key_iv: "771bffc1ef239e5b99c1cbe97493ed04"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.813575 5452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:07.856369 5452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:07.861282 5452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:07.861608 5452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:07.862783 5452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:07.864404 5452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:07.864491 5452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.864603 5452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:07.864660 5452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.894349 5313 heartbeater.cc:503] Master 127.4.243.254:45883 was elected leader, sending a full tablet report...
I20260925 14:58:07.900490 5452 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:34093
I20260925 14:58:07.900563 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:34093 every 8 connection(s)
I20260925 14:58:07.901746 5452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:07.903996 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5452
I20260925 14:58:07.904242 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:07.904825 5071 external_mini_cluster.cc:1428] Setting key 5b8beb8d856d5acd6ea3b083132c8577
Sep 25 14:58:07 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:07.923645 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.904952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:34021 (local address 127.4.243.254:45883)
0925 14:58:07.905427 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.905432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.906487 (+ 1055us) server_negotiation.cc:407] Connection header received
0925 14:58:07.908561 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.908566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.908650 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.908776 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.910171 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.911000 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.912465 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.912864 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.912969 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.916690 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.916695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.916698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.916727 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.919674 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.920668 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.920673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.920677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.920833 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.921397 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.921405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.921408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.921804 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.921956 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.923078 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.923280 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260925 14:58:07.924787 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.905580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45883 (local address 127.4.243.195:34021)
0925 14:58:07.906334 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.906380 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.908267 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.908969 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.908978 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.909462 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:07.910014 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.910031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.911120 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.911125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.912325 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.912336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.913052 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.913057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.913255 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.913991 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:07.914013 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:07.916479 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:07.919871 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.919880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.919897 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.920502 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.920982 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.920988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.920992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.921229 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.921942 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:07.921950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:07.922262 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.922520 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.923443 (+ 923us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":297,"threads_started":1}
I20260925 14:58:07.927436 5580 heartbeater.cc:347] Connected to a master server at 127.4.243.254:45883
I20260925 14:58:07.927742 5580 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:07.928442 5580 heartbeater.cc:511] Master 127.4.243.254:45883 requested a full tablet report, sending...
I20260925 14:58:07.929821 5123 ts_manager.cc:194] Registered new tserver with Master: 94423c72d1e54700ab73199acbe63dd2 (127.4.243.195:34093)
I20260925 14:58:07.930696 5123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:34021
I20260925 14:58:07.933907 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:58:07 dist-test-slave-qxh0 krb5kdc[5078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:07 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:07.979056 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.962978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.4.243.193:33323)
0925 14:58:07.963793 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.963801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.963833 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:58:07.963913 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.963921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.964158 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.964291 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.965629 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.966733 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.967970 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.969118 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.969227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.973247 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.973257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.973273 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.973305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.976241 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.976908 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.976915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.976917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.977038 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.977455 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.977462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.977465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.977805 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.978104 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.978403 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.978704 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":441,"threads_started":1}
W20260925 14:58:07.980423 5228 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55184
Sep 25 14:58:08 dist-test-slave-qxh0 krb5kdc[5078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:08 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:08.023128 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.009608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.4.243.193:33323)
0925 14:58:08.009885 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.009892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.010080 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:58:08.010248 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.010252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.010320 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.010405 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:08.012180 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.012839 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.013988 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.014378 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.014467 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.018339 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:08.018344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:08.018347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:08.018379 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:08.020903 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.021471 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.021475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.021477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.021585 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.021939 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.021942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.021944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.022165 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:08.022329 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.022569 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.022800 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 14:58:08.024565 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5185
I20260925 14:58:08.036254 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.025596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.4.243.193:33323)
0925 14:58:08.025748 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.025751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.025981 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:58:08.026107 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.026110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.026162 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.026235 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:08.027366 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.028039 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.029272 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.029692 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.029779 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.031897 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:08.031901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:08.031904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:08.031932 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:08.034153 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.034717 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.034720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.034722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.034837 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.035215 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.035218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.035220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.035439 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:08.035601 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.035846 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.036003 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:58:08.036952 5308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:08.119628 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5185
I20260925 14:58:08.135996 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5319
Sep 25 14:58:08 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:08.151692 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.136903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.4.243.194:35931)
0925 14:58:08.137398 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.137405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.137429 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:58:08.137487 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.137493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.137710 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.137885 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:08.139160 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.140144 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.141357 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.141898 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.142044 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.146128 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:08.146140 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:08.146154 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:08.146205 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:08.149107 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.149744 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.149751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.149753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.149875 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.150223 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.150226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.150228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.150558 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:08.150854 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.151120 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.151310 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":266,"threads_started":1}
I20260925 14:58:08.152985 5442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:08.234941 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5319
I20260925 14:58:08.257118 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5452
Sep 25 14:58:08 dist-test-slave-qxh0 krb5kdc[5078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:08.272974 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.258013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.4.243.195:34093)
0925 14:58:08.258716 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.258723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.258753 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:58:08.258813 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.258819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.259028 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.259163 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:08.260337 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.261372 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.262655 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.263192 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.263297 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.267316 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:08.267327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:08.267341 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:08.267375 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:08.270159 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.270942 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.270950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.270952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.271071 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.271443 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.271450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.271453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.271780 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:08.272087 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.272383 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.272646 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":331,"threads_started":1}
I20260925 14:58:08.274257 5575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:08.355597 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5452
I20260925 14:58:08.371773 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5091
I20260925 14:58:08.372960 5153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:08.494985 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5091
2026-09-25T14:58:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3978 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:08 dist-test-slave-qxh0 krb5kdc[5602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:08 dist-test-slave-qxh0 krb5kdc[5602](info): set up 2 sockets
Sep 25 14:58:08 dist-test-slave-qxh0 krb5kdc[5602](info): commencing operation
krb5kdc: starting...
W20260925 14:58:09.566574 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:09 dist-test-slave-qxh0 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:09.742378 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43793
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:43793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:10.060034 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:10.060334 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:10.060520 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:10.068034 5615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:10.068149 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:10.068233 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:10.068331 5615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:10.068395 5615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:10.078536 5615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:43793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:10.079905 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:10.081270 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:10.092994 5620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:10.093222 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:10.093108 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:10.094235 5615 server_base.cc:1034] running on GCE node
I20260925 14:58:10.095168 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:10.096619 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:10.097815 5615 hybrid_clock.cc:648] HybridClock initialized: now 1790348290097745 us; error 62 us; skew 500 ppm
I20260925 14:58:10.098163 5615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:10 dist-test-slave-qxh0 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348290, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:10.107553 5615 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:10.109185 5615 webserver.cc:460] Webserver started at http://127.4.243.254:39657/ using document root <none> and password file <none>
I20260925 14:58:10.109903 5615 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:10.110031 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:10.110363 5615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:10.113399 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "ca6abc07b6b1458ebf0cf2f53ea772de"
format_stamp: "Formatted at 2026-09-25 14:58:10 on dist-test-slave-qxh0"
server_key: "0fde9a880b3358ff2991b7816781bd14"
server_key_iv: "914b5673eb14c4450d767ae867d28356"
server_key_version: "encryptionkey@0"
I20260925 14:58:10.114187 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "ca6abc07b6b1458ebf0cf2f53ea772de"
format_stamp: "Formatted at 2026-09-25 14:58:10 on dist-test-slave-qxh0"
server_key: "0fde9a880b3358ff2991b7816781bd14"
server_key_iv: "914b5673eb14c4450d767ae867d28356"
server_key_version: "encryptionkey@0"
I20260925 14:58:10.119484 5615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 14:58:10.123013 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:10.124351 5615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:58:10.124522 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "ca6abc07b6b1458ebf0cf2f53ea772de"
format_stamp: "Formatted at 2026-09-25 14:58:10 on dist-test-slave-qxh0"
server_key: "0fde9a880b3358ff2991b7816781bd14"
server_key_iv: "914b5673eb14c4450d767ae867d28356"
server_key_version: "encryptionkey@0"
I20260925 14:58:10.124739 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:10.159169 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:10.163820 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:10.164158 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:10.181849 5615 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:43793
I20260925 14:58:10.181854 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:43793 every 8 connection(s)
I20260925 14:58:10.183447 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:10.187318 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:10.189740 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5615
I20260925 14:58:10.189970 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:10.190773 5071 external_mini_cluster.cc:1428] Setting key 25f4b0a2211972d503bb9dab4dab973e
I20260925 14:58:10.199931 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de: Bootstrap starting.
I20260925 14:58:10.203792 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:10.205113 5682 log.cc:824] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:58:10 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:10.208431 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de: No bootstrap required, opened a new log
I20260925 14:58:10.214794 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.192918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.4.243.254:43793)
0925 14:58:10.193737 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.193749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.193791 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:58:10.195500 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.195524 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.195926 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.196262 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:10.197961 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.199336 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.201017 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.201713 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.201843 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.206896 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:10.206903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:10.206920 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:10.206954 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:10.210273 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.211089 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.211096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.211103 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.211278 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.211667 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.211678 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.211683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.212268 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:10.212615 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.213153 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.213592 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":391,"threads_started":1}
I20260925 14:58:10.216816 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43793 } }
I20260925 14:58:10.217284 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:10.217389 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6abc07b6b1458ebf0cf2f53ea772de, State: Initialized, Role: FOLLOWER
I20260925 14:58:10.218031 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43793 } }
I20260925 14:58:10.218297 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:10.218400 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:10.218515 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:10.220953 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43793 } }
I20260925 14:58:10.221589 5682 leader_election.cc:302] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6abc07b6b1458ebf0cf2f53ea772de; no voters:
I20260925 14:58:10.222198 5682 leader_election.cc:288] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:10.222388 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:10.223088 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [term 1 LEADER]: Becoming Leader. State: Replica: ca6abc07b6b1458ebf0cf2f53ea772de, State: Running, Role: LEADER
I20260925 14:58:10.223721 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43793 } }
I20260925 14:58:10.224368 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:10.227125 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6abc07b6b1458ebf0cf2f53ea772de. Latest consensus state: current_term: 1 leader_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43793 } } }
I20260925 14:58:10.227597 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:10.227944 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6abc07b6b1458ebf0cf2f53ea772de" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43793 } } }
I20260925 14:58:10.228294 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:10.229538 5696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:10.234691 5696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:10.244304 5696 catalog_manager.cc:1347] Generated new cluster ID: 98719c75a29e4cb5890bb90742e580a9
I20260925 14:58:10.244441 5696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:10.256897 5696 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:10.258956 5696 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:10.270720 5696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6abc07b6b1458ebf0cf2f53ea772de: Generated new TSK 0
I20260925 14:58:10.271840 5696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:10.320672 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43793
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:10.645869 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:10.646147 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:10.646309 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:10.653573 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:10.653729 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:10.653914 5708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:10.663744 5708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:10.665076 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:10.666395 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:10.679147 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:10.679195 5716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:10.679224 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:10.680161 5708 server_base.cc:1034] running on GCE node
I20260925 14:58:10.680738 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:10.681641 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:10.682876 5708 hybrid_clock.cc:648] HybridClock initialized: now 1790348290682819 us; error 39 us; skew 500 ppm
I20260925 14:58:10.683197 5708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:10 dist-test-slave-qxh0 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348290, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:10.686326 5708 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:10.687855 5708 webserver.cc:460] Webserver started at http://127.4.243.193:39755/ using document root <none> and password file <none>
I20260925 14:58:10.688544 5708 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:10.688683 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:10.688985 5708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:10.691937 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "88ffa5201e3b4493b6ed52d356bc2a55"
format_stamp: "Formatted at 2026-09-25 14:58:10 on dist-test-slave-qxh0"
server_key: "f6333baf788f666a1389bfb72f1cce04"
server_key_iv: "7c43eab30fd79b3913fd607968ab7506"
server_key_version: "encryptionkey@0"
I20260925 14:58:10.692739 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "88ffa5201e3b4493b6ed52d356bc2a55"
format_stamp: "Formatted at 2026-09-25 14:58:10 on dist-test-slave-qxh0"
server_key: "f6333baf788f666a1389bfb72f1cce04"
server_key_iv: "7c43eab30fd79b3913fd607968ab7506"
server_key_version: "encryptionkey@0"
I20260925 14:58:10.697795 5708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:58:10.701157 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:10.702414 5708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:10.702629 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "88ffa5201e3b4493b6ed52d356bc2a55"
format_stamp: "Formatted at 2026-09-25 14:58:10 on dist-test-slave-qxh0"
server_key: "f6333baf788f666a1389bfb72f1cce04"
server_key_iv: "7c43eab30fd79b3913fd607968ab7506"
server_key_version: "encryptionkey@0"
I20260925 14:58:10.702839 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:10.715497 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:10.719823 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:10.720117 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:10.721060 5708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:10.722357 5708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:10.722458 5708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:10.722697 5708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:10.722774 5708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:10.755182 5708 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:36523
I20260925 14:58:10.755230 5835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:36523 every 8 connection(s)
I20260925 14:58:10.756505 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:10.758575 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5708
I20260925 14:58:10.758822 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:10.759553 5071 external_mini_cluster.cc:1428] Setting key dc19118552a54c4039a3959d0536e42e
Sep 25 14:58:10 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348290, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 14:58:10.780782 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.759372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:52509 (local address 127.4.243.254:43793)
0925 14:58:10.759845 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.759852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.760995 (+ 1143us) server_negotiation.cc:407] Connection header received
0925 14:58:10.763719 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.763723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.763782 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.763874 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:10.766007 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.766981 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.768546 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.769304 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.769444 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.773839 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:10.773848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:10.773856 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:10.773903 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:10.777422 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.778219 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.778224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.778226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.778343 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.778906 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.778914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.778919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.779204 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:10.779369 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.780188 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.780399 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260925 14:58:10.781788 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.759967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43793 (local address 127.4.243.193:52509)
0925 14:58:10.760838 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:10.760876 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:10.763395 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:10.764164 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:10.764181 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:10.765001 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:10.765814 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.765841 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.767165 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.767169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.768393 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.768408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.769502 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.769506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.769820 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.770632 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:10.770670 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:10.773566 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:10.777577 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:10.777587 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:10.777603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:10.778107 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:10.778467 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:10.778471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:10.778473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:10.778782 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:10.779344 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:10.779352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:10.779763 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 14:58:10.780063 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 14:58:10.780637 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":347,"threads_started":1}
I20260925 14:58:10.783448 5836 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43793
I20260925 14:58:10.783797 5836 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:10.784693 5836 heartbeater.cc:511] Master 127.4.243.254:43793 requested a full tablet report, sending...
I20260925 14:58:10.786861 5647 ts_manager.cc:194] Registered new tserver with Master: 88ffa5201e3b4493b6ed52d356bc2a55 (127.4.243.193:36523)
I20260925 14:58:10.788292 5647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:52509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:10.802076 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43793
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:11.120595 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:11.120875 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:11.121029 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:11.128307 5842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:11.128469 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:11.128604 5842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:11.138227 5842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:11.139475 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:11.140795 5842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:11.153827 5847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:11.153791 5848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:11.153954 5850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:11.154839 5842 server_base.cc:1034] running on GCE node
I20260925 14:58:11.155473 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:11.156383 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:11.157582 5842 hybrid_clock.cc:648] HybridClock initialized: now 1790348291157536 us; error 48 us; skew 500 ppm
I20260925 14:58:11.157891 5842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:11 dist-test-slave-qxh0 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:11.160909 5842 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:11.162359 5842 webserver.cc:460] Webserver started at http://127.4.243.194:45127/ using document root <none> and password file <none>
I20260925 14:58:11.163097 5842 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:11.163237 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:11.163568 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:11.166335 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "2c294f38a8b54b34b2c27ac1b3c3c288"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-qxh0"
server_key: "b609492a295e9ed5d0e60fe98426f25e"
server_key_iv: "260aa977700545d6ea2bcfbc27422e46"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.167130 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "2c294f38a8b54b34b2c27ac1b3c3c288"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-qxh0"
server_key: "b609492a295e9ed5d0e60fe98426f25e"
server_key_iv: "260aa977700545d6ea2bcfbc27422e46"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.172268 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:58:11.175658 5857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:11.176963 5842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:58:11.177090 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "2c294f38a8b54b34b2c27ac1b3c3c288"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-qxh0"
server_key: "b609492a295e9ed5d0e60fe98426f25e"
server_key_iv: "260aa977700545d6ea2bcfbc27422e46"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.177251 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:11.204648 5842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:11.208673 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:11.208900 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:11.209757 5842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:11.210978 5842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:11.211061 5842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:11.211162 5842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:11.211205 5842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:11.243180 5842 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:43613
I20260925 14:58:11.243253 5969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:43613 every 8 connection(s)
I20260925 14:58:11.244431 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:11.249061 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5842
I20260925 14:58:11.249260 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:11.249701 5071 external_mini_cluster.cc:1428] Setting key 9c2363000374b4fffacc25c3ae0cd874
Sep 25 14:58:11 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:11.264935 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.247012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:60603 (local address 127.4.243.254:43793)
0925 14:58:11.247187 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:11.247192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:11.248362 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 14:58:11.250411 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:11.250418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:11.250518 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:11.250667 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:11.252098 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.252796 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.254331 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.254795 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.254920 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.258811 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:11.258815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:11.258819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:11.258848 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:11.261395 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:11.262257 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:11.262266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:11.262271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:11.262439 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:11.262968 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:11.262975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:11.262980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:11.263392 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:11.263543 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:11.264326 (+ 783us) server_negotiation.cc:299] Negotiation successful
0925 14:58:11.264562 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:11.265727 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.247507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43793 (local address 127.4.243.194:60603)
0925 14:58:11.248208 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:11.248245 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:11.250137 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:11.250886 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:11.250895 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:11.251374 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:11.251926 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.251943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.252947 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.252958 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.254183 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.254194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.255017 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.255023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.255248 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.255921 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:11.255942 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:11.258554 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:11.261535 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:11.261544 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:11.261561 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:11.262135 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:11.262582 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:11.262585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:11.262588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:11.262849 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:11.263610 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:11.263616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:11.263932 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 14:58:11.264159 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 14:58:11.264556 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":279,"threads_started":1}
I20260925 14:58:11.267194 5970 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43793
I20260925 14:58:11.267511 5970 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 14:58:11.268316 5970 heartbeater.cc:511] Master 127.4.243.254:43793 requested a full tablet report, sending...
I20260925 14:58:11.269881 5647 ts_manager.cc:194] Registered new tserver with Master: 2c294f38a8b54b34b2c27ac1b3c3c288 (127.4.243.194:43613)
I20260925 14:58:11.270766 5647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:60603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:11.287894 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43793
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:11.602336 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:11.602581 5975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:11.602694 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:11.610739 5975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:11.610833 5975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:11.610952 5975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:11.620720 5975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:11.621872 5975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:11.623195 5975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:11.636058 5980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:11.636088 5981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:11.636231 5983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:11.636778 5975 server_base.cc:1034] running on GCE node
I20260925 14:58:11.637378 5975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:11.638266 5975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:11.639518 5975 hybrid_clock.cc:648] HybridClock initialized: now 1790348291639454 us; error 59 us; skew 500 ppm
I20260925 14:58:11.639853 5975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:11 dist-test-slave-qxh0 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:11.642997 5975 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:11.644208 5975 webserver.cc:460] Webserver started at http://127.4.243.195:37487/ using document root <none> and password file <none>
I20260925 14:58:11.644794 5975 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:11.644886 5975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:11.645114 5975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:11.647850 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-qxh0"
server_key: "7f493bc6c5fa503fea11c629da6f7f44"
server_key_iv: "1b080accf755b5fa7c3f4ae820ea3809"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.648552 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-qxh0"
server_key: "7f493bc6c5fa503fea11c629da6f7f44"
server_key_iv: "1b080accf755b5fa7c3f4ae820ea3809"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.653505 5975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 14:58:11.656873 5990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:11.658231 5975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:58:11.658383 5975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-qxh0"
server_key: "7f493bc6c5fa503fea11c629da6f7f44"
server_key_iv: "1b080accf755b5fa7c3f4ae820ea3809"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.658589 5975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:11.678416 5975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:11.682828 5975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:11.683135 5975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:11.684043 5975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:11.685341 5975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:11.685411 5975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:11.685576 5975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:11.685626 5975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:11.718108 5975 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:46179
I20260925 14:58:11.718161 6102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:46179 every 8 connection(s)
I20260925 14:58:11.719417 5975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:11.723265 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 5975
I20260925 14:58:11.723445 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:11.723897 5071 external_mini_cluster.cc:1428] Setting key 556311ecefd07a15c03bec03f045556e
Sep 25 14:58:11 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:11.739495 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.721970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:59405 (local address 127.4.243.254:43793)
0925 14:58:11.722144 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:11.722149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:11.723320 (+ 1171us) server_negotiation.cc:407] Connection header received
0925 14:58:11.725344 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:11.725349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:11.725436 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:11.725545 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:11.727054 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.727778 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.729378 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.729797 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.729925 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.733604 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:11.733612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:11.733622 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:11.733652 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:11.736558 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:11.737260 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:11.737265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:11.737271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:11.737382 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:11.737803 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:11.737807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:11.737812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:11.738079 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:11.738226 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:11.738935 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 14:58:11.739143 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:58:11.740438 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.722447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43793 (local address 127.4.243.195:59405)
0925 14:58:11.723162 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:11.723204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:11.725059 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:11.725782 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:11.725790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:11.726265 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:11.726887 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.726905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.727945 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.727948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.729246 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.729258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.730024 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.730031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.730270 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.731137 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:11.731162 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:11.733398 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:11.736699 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:11.736708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:11.736724 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:11.737133 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:11.737493 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:11.737499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:11.737505 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:11.737684 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:11.738203 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:11.738211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:11.738535 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 14:58:11.738852 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 14:58:11.739370 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":340,"threads_started":1}
I20260925 14:58:11.741724 6103 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43793
I20260925 14:58:11.742004 6103 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:11.742707 6103 heartbeater.cc:511] Master 127.4.243.254:43793 requested a full tablet report, sending...
I20260925 14:58:11.743994 5647 ts_manager.cc:194] Registered new tserver with Master: bb4d160bb83d4b1fb2db8d7e7da31b80 (127.4.243.195:46179)
I20260925 14:58:11.744737 5647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:59405
I20260925 14:58:11.750902 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:11.769793 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.756881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.4.243.254:43793)
0925 14:58:11.757078 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:11.757085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:11.757648 (+ 563us) server_negotiation.cc:407] Connection header received
0925 14:58:11.757772 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:11.757778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:11.757849 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:11.757953 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:11.759404 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.760592 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.761926 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.762338 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.762432 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.764939 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:11.764944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:11.764948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:11.764978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:11.767650 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:11.768241 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:11.768245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:11.768248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:11.768370 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:11.768659 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:11.768662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:11.768664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:11.768935 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:11.769124 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:11.769403 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:58:11.769567 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:58:11.776242 5647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:11.783125 5647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:11.791982 5836 heartbeater.cc:503] Master 127.4.243.254:43793 was elected leader, sending a full tablet report...
I20260925 14:58:11.808655 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.795461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:36523 (local address 127.0.0.1:54932)
0925 14:58:11.796273 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:11.796338 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:11.796576 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:11.798106 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:11.798117 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:11.798159 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:11.798666 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.798681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.801188 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.801197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.804382 (+ 3185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.804406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.807262 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.807271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.807569 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.807661 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 14:58:11.807796 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 14:58:11.808061 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":705,"threads_started":1}
I20260925 14:58:11.811232 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.795966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.4.243.193:36523)
0925 14:58:11.797581 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:11.797587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:11.797610 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:58:11.797686 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:11.797692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:11.797880 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:11.798038 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:11.798867 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.800971 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.804655 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.806979 (+ 2324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.810125 (+ 3146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.810436 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:11.810523 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 14:58:11.810798 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1474,"thread_start_us":1357,"threads_started":1}
I20260925 14:58:11.811250 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.795693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43613 (local address 127.0.0.1:48106)
0925 14:58:11.798568 (+ 2875us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:11.798587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:11.798776 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:11.799807 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:11.799816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:11.799852 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:11.800441 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.800466 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.803156 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.803164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.806897 (+ 3733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.806917 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.810028 (+ 3111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.810038 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.810326 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.810383 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 14:58:11.810512 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:58:11.810745 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2746,"thread_start_us":2601,"threads_started":1}
I20260925 14:58:11.812288 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.795906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.4.243.194:43613)
0925 14:58:11.797182 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:11.797190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:11.798908 (+ 1718us) server_negotiation.cc:407] Connection header received
0925 14:58:11.799024 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:11.799032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:11.799281 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:11.799527 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:11.800688 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.802912 (+ 2224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.807160 (+ 4248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.809342 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.809543 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.809829 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:11.811583 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0925 14:58:11.811834 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":1051,"threads_started":1}
I20260925 14:58:11.816133 5771 tablet_service.cc:1467] Processing CreateTablet for tablet d1b907638edc48708d1ee9876bf80930 (DEFAULT_TABLE table=test-table [id=3d1014e4c5c34ab1acf0be94d13ccd6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:11.817689 5771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b907638edc48708d1ee9876bf80930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:11.817196 5905 tablet_service.cc:1467] Processing CreateTablet for tablet d1b907638edc48708d1ee9876bf80930 (DEFAULT_TABLE table=test-table [id=3d1014e4c5c34ab1acf0be94d13ccd6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:11.818528 5905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b907638edc48708d1ee9876bf80930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:11.820622 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.798746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:46179 (local address 127.0.0.1:36060)
0925 14:58:11.809754 (+ 11008us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:11.809777 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:11.809906 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:11.814096 (+ 4190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:11.814101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:11.814126 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:11.814313 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.814322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.816157 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.816163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.817707 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:11.817720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.819724 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.819730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:11.819932 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.819968 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:58:11.820104 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:58:11.820268 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10857,"thread_start_us":14738,"threads_started":1}
I20260925 14:58:11.820640 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:11.799100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.4.243.195:46179)
0925 14:58:11.813367 (+ 14267us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:11.813379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:11.813421 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:58:11.813550 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:11.813560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:11.813858 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:11.814472 (+ 614us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:11.814806 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.816008 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.817895 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:11.819553 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:11.819712 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:11.819967 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:11.820143 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 14:58:11.820305 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14093,"thread_start_us":13914,"threads_started":1}
I20260925 14:58:11.824608 6038 tablet_service.cc:1467] Processing CreateTablet for tablet d1b907638edc48708d1ee9876bf80930 (DEFAULT_TABLE table=test-table [id=3d1014e4c5c34ab1acf0be94d13ccd6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:11.825970 6038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1b907638edc48708d1ee9876bf80930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:11.833117 6121 tablet_bootstrap.cc:492] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55: Bootstrap starting.
I20260925 14:58:11.835489 6122 tablet_bootstrap.cc:492] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288: Bootstrap starting.
I20260925 14:58:11.837045 6121 tablet_bootstrap.cc:654] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:11.838466 6121 log.cc:824] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:11.839306 6122 tablet_bootstrap.cc:654] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:11.840353 6122 log.cc:824] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:11.841820 6121 tablet_bootstrap.cc:492] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55: No bootstrap required, opened a new log
I20260925 14:58:11.842134 6121 ts_tablet_manager.cc:1397] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 14:58:11.843030 6122 tablet_bootstrap.cc:492] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288: No bootstrap required, opened a new log
I20260925 14:58:11.843276 6122 ts_tablet_manager.cc:1397] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 14:58:11.843278 6125 tablet_bootstrap.cc:492] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80: Bootstrap starting.
I20260925 14:58:11.847295 6125 tablet_bootstrap.cc:654] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:11.848587 6125 log.cc:824] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:11.851802 6125 tablet_bootstrap.cc:492] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80: No bootstrap required, opened a new log
I20260925 14:58:11.851775 6122 raft_consensus.cc:348] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:11.852087 6125 ts_tablet_manager.cc:1397] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260925 14:58:11.852268 6122 raft_consensus.cc:374] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:11.852396 6122 raft_consensus.cc:729] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c294f38a8b54b34b2c27ac1b3c3c288, State: Initialized, Role: FOLLOWER
I20260925 14:58:11.852993 6122 consensus_queue.cc:260] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:11.853400 6121 raft_consensus.cc:348] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:11.853909 6121 raft_consensus.cc:374] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:11.854067 6121 raft_consensus.cc:729] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ffa5201e3b4493b6ed52d356bc2a55, State: Initialized, Role: FOLLOWER
I20260925 14:58:11.854839 6121 consensus_queue.cc:260] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:11.855758 5970 heartbeater.cc:503] Master 127.4.243.254:43793 was elected leader, sending a full tablet report...
I20260925 14:58:11.856463 6122 ts_tablet_manager.cc:1428] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260925 14:58:11.858152 6121 ts_tablet_manager.cc:1428] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260925 14:58:11.864250 6125 raft_consensus.cc:348] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:11.864669 6125 raft_consensus.cc:374] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:11.864777 6125 raft_consensus.cc:729] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4d160bb83d4b1fb2db8d7e7da31b80, State: Initialized, Role: FOLLOWER
I20260925 14:58:11.865450 6125 consensus_queue.cc:260] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:11.866501 6103 heartbeater.cc:503] Master 127.4.243.254:43793 was elected leader, sending a full tablet report...
I20260925 14:58:11.867308 6125 ts_tablet_manager.cc:1428] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
W20260925 14:58:11.971004 6104 tablet.cc:2369] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:11.996430 5971 tablet.cc:2369] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:12.008985 5837 tablet.cc:2369] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:12.107301 6129 raft_consensus.cc:482] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:12.107649 6129 raft_consensus.cc:504] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:12.109357 6129 leader_election.cc:288] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88ffa5201e3b4493b6ed52d356bc2a55 (127.4.243.193:36523), 2c294f38a8b54b34b2c27ac1b3c3c288 (127.4.243.194:43613)
I20260925 14:58:12.112478 6128 raft_consensus.cc:482] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:12.112757 6128 raft_consensus.cc:504] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:12.114543 6128 leader_election.cc:288] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c294f38a8b54b34b2c27ac1b3c3c288 (127.4.243.194:43613), bb4d160bb83d4b1fb2db8d7e7da31b80 (127.4.243.195:46179)
I20260925 14:58:12.117631 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.109576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43613 (local address 127.4.243.195:59607)
0925 14:58:12.109760 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.109785 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.109934 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.111095 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.111099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.111125 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.111299 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.111307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.112696 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.112701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.114298 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.114310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.116564 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.116574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.116836 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.116888 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.116994 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.117179 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":55}
I20260925 14:58:12.118434 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.109612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:46533 (local address 127.4.243.193:36523)
0925 14:58:12.109868 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.109876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.110972 (+ 1096us) server_negotiation.cc:407] Connection header received
0925 14:58:12.111221 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.111228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.111295 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.111365 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.111825 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.112848 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.115292 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.116990 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.117154 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.117258 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.117931 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.118110 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 14:58:12.118535 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.110485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:59607 (local address 127.4.243.194:43613)
0925 14:58:12.110741 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.110747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.110768 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:58:12.110835 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.110839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.110892 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.111047 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.111457 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.112546 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.114513 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.116145 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.116338 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.116456 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.118002 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.118185 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 14:58:12.119244 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.109583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:36523 (local address 127.4.243.195:46533)
0925 14:58:12.110850 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.110865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.110984 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.111415 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.111422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.111455 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.111693 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.111700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.113023 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.113028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.115104 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.115119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.117490 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.117499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.117735 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.117774 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.118579 (+ 805us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.118747 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"mutex_wait_us":29,"thread_start_us":863,"threads_started":1}
I20260925 14:58:12.119645 5925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b907638edc48708d1ee9876bf80930" candidate_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" is_pre_election: true
I20260925 14:58:12.120378 5925 raft_consensus.cc:2454] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4d160bb83d4b1fb2db8d7e7da31b80 in term 0.
I20260925 14:58:12.121297 5991 leader_election.cc:302] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c294f38a8b54b34b2c27ac1b3c3c288, bb4d160bb83d4b1fb2db8d7e7da31b80; no voters:
I20260925 14:58:12.121685 6129 raft_consensus.cc:2781] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:12.121826 6129 raft_consensus.cc:482] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:12.121933 6129 raft_consensus.cc:3037] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.122467 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b907638edc48708d1ee9876bf80930" candidate_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" is_pre_election: true
I20260925 14:58:12.122896 5791 raft_consensus.cc:2454] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4d160bb83d4b1fb2db8d7e7da31b80 in term 0.
I20260925 14:58:12.124773 6129 raft_consensus.cc:504] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:12.125377 6129 leader_election.cc:288] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [CANDIDATE]: Term 1 election: Requested vote from peers 88ffa5201e3b4493b6ed52d356bc2a55 (127.4.243.193:36523), 2c294f38a8b54b34b2c27ac1b3c3c288 (127.4.243.194:43613)
I20260925 14:58:12.125844 5791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b907638edc48708d1ee9876bf80930" candidate_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ffa5201e3b4493b6ed52d356bc2a55"
I20260925 14:58:12.126029 5791 raft_consensus.cc:3037] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.126008 5925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b907638edc48708d1ee9876bf80930" candidate_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288"
I20260925 14:58:12.126195 5925 raft_consensus.cc:3037] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.128593 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.115154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:48075 (local address 127.4.243.194:43613)
0925 14:58:12.119925 (+ 4771us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.119931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.120179 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:58:12.120457 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.120461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.120513 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.120792 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.121987 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.123132 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.125614 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.127102 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.127294 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.127366 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.128084 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.128226 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4599,"thread_start_us":4339,"threads_started":1}
I20260925 14:58:12.129202 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.114975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43613 (local address 127.4.243.193:48075)
0925 14:58:12.120012 (+ 5037us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.120031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.120276 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.120805 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.120809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.120842 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.121768 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.121782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.123305 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.123309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.125434 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.125449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.127624 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.127631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.127819 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.127851 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.128739 (+ 888us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.128867 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4908,"thread_start_us":4751,"threads_started":1}
I20260925 14:58:12.129493 5791 raft_consensus.cc:2454] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4d160bb83d4b1fb2db8d7e7da31b80 in term 1.
I20260925 14:58:12.129571 5925 raft_consensus.cc:2454] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4d160bb83d4b1fb2db8d7e7da31b80 in term 1.
I20260925 14:58:12.129971 5991 leader_election.cc:302] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c294f38a8b54b34b2c27ac1b3c3c288, bb4d160bb83d4b1fb2db8d7e7da31b80; no voters:
I20260925 14:58:12.130620 6129 raft_consensus.cc:2781] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:12.130679 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.120183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:46179 (local address 127.4.243.193:42811)
0925 14:58:12.122184 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.122203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.122359 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.122855 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.122859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.122881 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.123304 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.123313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.124753 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.124758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.126872 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.126887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.129883 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.129889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.130067 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.130097 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.130185 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.130302 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"thread_start_us":1007,"threads_started":1}
I20260925 14:58:12.131079 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.120283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:42811 (local address 127.4.243.195:46179)
0925 14:58:12.120483 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.120491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.122272 (+ 1781us) server_negotiation.cc:407] Connection header received
0925 14:58:12.122514 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.122519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.122579 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.122660 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.123494 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.124588 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.127118 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.128466 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.128639 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.128706 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.130649 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.130773 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:58:12.131115 6129 raft_consensus.cc:686] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [term 1 LEADER]: Becoming Leader. State: Replica: bb4d160bb83d4b1fb2db8d7e7da31b80, State: Running, Role: LEADER
I20260925 14:58:12.131114 5925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b907638edc48708d1ee9876bf80930" candidate_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" is_pre_election: true
I20260925 14:58:12.131510 5925 raft_consensus.cc:2379] T d1b907638edc48708d1ee9876bf80930 P 2c294f38a8b54b34b2c27ac1b3c3c288 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88ffa5201e3b4493b6ed52d356bc2a55 in current term 1: Already voted for candidate bb4d160bb83d4b1fb2db8d7e7da31b80 in this term.
I20260925 14:58:12.131657 6129 consensus_queue.cc:237] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } }
I20260925 14:58:12.132638 6058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1b907638edc48708d1ee9876bf80930" candidate_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" is_pre_election: true
I20260925 14:58:12.133538 5724 leader_election.cc:302] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88ffa5201e3b4493b6ed52d356bc2a55; no voters: 2c294f38a8b54b34b2c27ac1b3c3c288, bb4d160bb83d4b1fb2db8d7e7da31b80
I20260925 14:58:12.133961 6128 raft_consensus.cc:2726] T d1b907638edc48708d1ee9876bf80930 P 88ffa5201e3b4493b6ed52d356bc2a55 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 14:58:12.136724 5646 catalog_manager.cc:5495] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb4d160bb83d4b1fb2db8d7e7da31b80 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ffa5201e3b4493b6ed52d356bc2a55" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c294f38a8b54b34b2c27ac1b3c3c288" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb4d160bb83d4b1fb2db8d7e7da31b80" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 46179 } health_report { overall_health: HEALTHY } } }
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:12.197962 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.181363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.4.243.193:36523)
0925 14:58:12.181566 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.181572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.182148 (+ 576us) server_negotiation.cc:407] Connection header received
0925 14:58:12.182298 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.182302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.182366 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.182459 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.184243 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.184986 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.187277 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.187691 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.187793 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.192299 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.192311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.192326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.192356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.195382 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.196039 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.196046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.196048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.196170 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.196555 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.196559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.196562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.196877 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.197175 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.197445 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.197653 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:12.242761 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.229695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.4.243.193:36523)
0925 14:58:12.229951 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.229957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.230173 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:58:12.230321 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.230325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.230391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.230489 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.231832 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.232602 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.233738 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.234114 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.234201 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.237633 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.237637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.237640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.237667 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.240146 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.241004 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.241010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.241015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.241177 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.241549 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.241552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.241554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.241771 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.241955 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.242165 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.242376 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[5602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:12.284637 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.271484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.4.243.193:36523)
0925 14:58:12.271721 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.271728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.271967 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:58:12.272168 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.272173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.272232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.272334 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.273526 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.274194 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.275459 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.275891 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.276034 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.279702 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.279706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.279709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.279740 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.282421 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.283086 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.283090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.283092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.283203 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.283503 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.283506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.283508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.283774 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.283969 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.284136 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.284293 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 14:58:12.286240 5751 tablet_service.cc:2908] Rejecting scan request for tablet d1b907638edc48708d1ee9876bf80930: Uninitialized: clean time has not yet been initialized
I20260925 14:58:12.289072 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5708
I20260925 14:58:12.301793 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.290239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.4.243.193:36523)
0925 14:58:12.290423 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.290429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.290649 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:58:12.290842 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.290846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.290906 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.291011 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.292224 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.292870 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.294032 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.294445 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.294580 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.296647 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.296652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.296655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.296680 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.299522 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.300122 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.300128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.300136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.300237 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.300573 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.300580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.300583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.300818 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.301087 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.301330 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.301518 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:58:12.302495 5831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:12.390866 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5708
I20260925 14:58:12.408313 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5842
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:12.422482 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.409183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.4.243.194:43613)
0925 14:58:12.409349 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.409353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.409371 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:58:12.409511 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.409515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.409584 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.409665 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.410833 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.411496 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.412705 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.413084 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.413176 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.417142 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.417156 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.417172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.417204 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.420070 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.420681 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.420687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.420689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.420811 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.421160 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.421163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.421165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.421490 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.421712 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.421944 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.422103 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:58:12.423156 5965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:12.511006 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5842
W20260925 14:58:12.526803 5991 connection.cc:535] client connection to 127.4.243.194:43613 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:58:12.527093 5991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:58:12.527395 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5975
I20260925 14:58:12.528690 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.528253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43613 (local address 127.4.243.195:44897)
0925 14:58:12.528415 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.528529 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.194:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 14:58:12.528932 5991 consensus_peers.cc:487] T d1b907638edc48708d1ee9876bf80930 P bb4d160bb83d4b1fb2db8d7e7da31b80 -> Peer 2c294f38a8b54b34b2c27ac1b3c3c288 (127.4.243.194:43613): Couldn't send request to peer 2c294f38a8b54b34b2c27ac1b3c3c288. Status: Network error: Client connection negotiation failed: client connection to 127.4.243.194:43613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[5602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:12.542187 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.528450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.4.243.195:46179)
0925 14:58:12.528597 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.528602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.528619 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:58:12.528682 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.528690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.528799 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.528939 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.530123 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.530973 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532235 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.532604 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532738 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.536781 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.536792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.536806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.536835 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.539624 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.540260 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.540266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.540269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.540380 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.540754 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.540760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.540764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.541123 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.541405 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.541672 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.541873 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:58:12.542989 6098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:12.630445 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5975
I20260925 14:58:12.646287 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 5615
I20260925 14:58:12.647444 5677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:12.750072 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 5615
2026-09-25T14:58:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4254 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[6154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[6154](info): set up 2 sockets
Sep 25 14:58:12 dist-test-slave-qxh0 krb5kdc[6154](info): commencing operation
krb5kdc: starting...
W20260925 14:58:13.810494 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:13 dist-test-slave-qxh0 krb5kdc[6154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:13.986452 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44487
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:44487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:14.294049 6167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:14.294320 6167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:14.294487 6167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:14.301420 6167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:14.301507 6167 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:58:14.301596 6167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:14.301712 6167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:14.301791 6167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:14.301877 6167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:14.312162 6167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:44487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:14.313401 6167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:14.314697 6167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:14.325882 6175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:14.325927 6173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:14.326073 6172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:14.326879 6167 server_base.cc:1034] running on GCE node
I20260925 14:58:14.327646 6167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:14.328943 6167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:14.330209 6167 hybrid_clock.cc:648] HybridClock initialized: now 1790348294330123 us; error 54 us; skew 500 ppm
I20260925 14:58:14.330547 6167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:14 dist-test-slave-qxh0 krb5kdc[6154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:14.333501 6167 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:14.334995 6167 webserver.cc:460] Webserver started at http://127.4.243.254:34275/ using document root <none> and password file <none>
I20260925 14:58:14.335678 6167 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:14.335808 6167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:14.336117 6167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:14.339023 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "8fb64934ec954b689be2be362e968210"
format_stamp: "Formatted at 2026-09-25 14:58:14 on dist-test-slave-qxh0"
server_key: "60842b7a6087f37df672355b85b5bac0"
server_key_iv: "92298ad2b60ec614cbd7ac9c55dcace3"
server_key_version: "encryptionkey@0"
I20260925 14:58:14.339797 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "8fb64934ec954b689be2be362e968210"
format_stamp: "Formatted at 2026-09-25 14:58:14 on dist-test-slave-qxh0"
server_key: "60842b7a6087f37df672355b85b5bac0"
server_key_iv: "92298ad2b60ec614cbd7ac9c55dcace3"
server_key_version: "encryptionkey@0"
I20260925 14:58:14.344611 6167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:58:14.347908 6182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:14.349124 6167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:58:14.349299 6167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "8fb64934ec954b689be2be362e968210"
format_stamp: "Formatted at 2026-09-25 14:58:14 on dist-test-slave-qxh0"
server_key: "60842b7a6087f37df672355b85b5bac0"
server_key_iv: "92298ad2b60ec614cbd7ac9c55dcace3"
server_key_version: "encryptionkey@0"
I20260925 14:58:14.349500 6167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:14.368922 6167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:14.373196 6167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:14.373456 6167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:14.392989 6167 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:44487
I20260925 14:58:14.393006 6243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:44487 every 8 connection(s)
I20260925 14:58:14.394223 6167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:14.397889 6244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:14.401751 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 6167
I20260925 14:58:14.401974 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:14.402769 5071 external_mini_cluster.cc:1428] Setting key 4aae01504aadd957dc581f71af9f90ea
I20260925 14:58:14.409078 6244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210: Bootstrap starting.
I20260925 14:58:14.412422 6244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:14.413529 6244 log.cc:824] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:14.416828 6244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210: No bootstrap required, opened a new log
Sep 25 14:58:14 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:14.424197 6244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb64934ec954b689be2be362e968210" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44487 } }
I20260925 14:58:14.424655 6244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:14.424769 6244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb64934ec954b689be2be362e968210, State: Initialized, Role: FOLLOWER
I20260925 14:58:14.425361 6244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb64934ec954b689be2be362e968210" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44487 } }
I20260925 14:58:14.425596 6244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:14.425653 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.404771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.4.243.254:44487)
0925 14:58:14.405474 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.405486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.405528 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:58:14.407650 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.407679 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.408027 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.408368 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:14.410026 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.411335 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.413062 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.413792 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.413921 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.418577 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:14.418585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:14.418642 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:14.418696 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:14.421617 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.422251 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.422265 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.422270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.422435 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.422833 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.422837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.422839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.423330 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:14.423620 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.424025 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.424418 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":332,"threads_started":1}
I20260925 14:58:14.425762 6244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:14.425940 6244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:14.429054 6244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb64934ec954b689be2be362e968210" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44487 } }
I20260925 14:58:14.429528 6244 leader_election.cc:302] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fb64934ec954b689be2be362e968210; no voters:
I20260925 14:58:14.430081 6244 leader_election.cc:288] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:14.430220 6249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:14.430965 6249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [term 1 LEADER]: Becoming Leader. State: Replica: 8fb64934ec954b689be2be362e968210, State: Running, Role: LEADER
I20260925 14:58:14.431528 6249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb64934ec954b689be2be362e968210" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44487 } }
I20260925 14:58:14.432081 6244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:14.434391 6250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fb64934ec954b689be2be362e968210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb64934ec954b689be2be362e968210" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44487 } } }
I20260925 14:58:14.434759 6250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:14.436908 6254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:14.436769 6251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb64934ec954b689be2be362e968210. Latest consensus state: current_term: 1 leader_uuid: "8fb64934ec954b689be2be362e968210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb64934ec954b689be2be362e968210" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44487 } } }
I20260925 14:58:14.437081 6251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:14.441660 6254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:14.449378 6254 catalog_manager.cc:1347] Generated new cluster ID: db7bef8762d64e8aad83d4e94fd13b68
I20260925 14:58:14.449519 6254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:14.464815 6254 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:14.466284 6254 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:14.472934 6254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210: Generated new TSK 0
I20260925 14:58:14.473713 6254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:14.513743 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44487
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:14.838832 6270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:14.839143 6270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:14.839315 6270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:14.846004 6270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:14.846093 6270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:14.846256 6270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:14.846369 6270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:14.855311 6270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:44487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:14.856460 6270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:14.857726 6270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:14.869903 6276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:14.869947 6278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:14.869973 6275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:14.870955 6270 server_base.cc:1034] running on GCE node
I20260925 14:58:14.871534 6270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:14.872399 6270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:14.873624 6270 hybrid_clock.cc:648] HybridClock initialized: now 1790348294873557 us; error 50 us; skew 500 ppm
I20260925 14:58:14.873912 6270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:14 dist-test-slave-qxh0 krb5kdc[6154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:14.876879 6270 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:14.878096 6270 webserver.cc:460] Webserver started at http://127.4.243.193:38073/ using document root <none> and password file <none>
I20260925 14:58:14.878800 6270 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:14.878906 6270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:14.879210 6270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:14.881836 6270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "e91f97e8f533437a8fe1ededb7891df1"
format_stamp: "Formatted at 2026-09-25 14:58:14 on dist-test-slave-qxh0"
server_key: "e3fca8f86390841a0282303499cc44bf"
server_key_iv: "8e33f0e84296227a4158b255ba978277"
server_key_version: "encryptionkey@0"
I20260925 14:58:14.882642 6270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "e91f97e8f533437a8fe1ededb7891df1"
format_stamp: "Formatted at 2026-09-25 14:58:14 on dist-test-slave-qxh0"
server_key: "e3fca8f86390841a0282303499cc44bf"
server_key_iv: "8e33f0e84296227a4158b255ba978277"
server_key_version: "encryptionkey@0"
I20260925 14:58:14.887257 6270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 14:58:14.890501 6285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:14.891770 6270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:58:14.891937 6270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "e91f97e8f533437a8fe1ededb7891df1"
format_stamp: "Formatted at 2026-09-25 14:58:14 on dist-test-slave-qxh0"
server_key: "e3fca8f86390841a0282303499cc44bf"
server_key_iv: "8e33f0e84296227a4158b255ba978277"
server_key_version: "encryptionkey@0"
I20260925 14:58:14.892138 6270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:14.913460 6270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:14.917662 6270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:14.917914 6270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:14.920434 6270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:14.920542 6270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:14.920691 6270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:14.920766 6270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:14 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:14.943861 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.921306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:59521 (local address 127.4.243.254:44487)
0925 14:58:14.921473 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.921479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.922803 (+ 1324us) server_negotiation.cc:407] Connection header received
0925 14:58:14.924953 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.924961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.925050 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.925126 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:14.927625 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.928677 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.931301 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.932001 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.932192 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.937736 (+ 5544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:14.937750 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:14.937757 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:14.937819 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:14.940987 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.941672 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.941679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.941683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.941801 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.942216 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.942220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.942224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.942479 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:14.942651 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.943323 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.943526 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:58:14.944880 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.921833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44487 (local address 127.4.243.193:59521)
0925 14:58:14.922636 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:14.922684 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:14.924598 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:14.925694 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:14.925710 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:14.926490 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:14.927384 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.927414 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.928828 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.928833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.931132 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.931148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.932316 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.932326 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.932700 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.933530 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:14.933586 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:14.937449 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:14.941131 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:14.941140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:14.941157 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:14.941544 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:14.941907 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:14.941915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:14.941918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:14.942101 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:14.942626 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:14.942634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:14.942987 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 14:58:14.943243 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 14:58:14.943763 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":505,"threads_started":1}
I20260925 14:58:14.968524 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.953304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:56819 (local address 127.4.243.254:44487)
0925 14:58:14.953453 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.953458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.954738 (+ 1280us) server_negotiation.cc:407] Connection header received
0925 14:58:14.954980 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.954989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.955081 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.955203 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:14.956137 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.957574 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.958904 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.959538 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.959699 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.962246 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:14.962251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:14.962255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:14.962282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:14.965523 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.966500 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.966509 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.966513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.966712 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.967084 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.967091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.967094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.967350 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:14.967509 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.967951 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.968171 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:58:14.968688 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.953446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44487 (local address 127.4.243.193:56819)
0925 14:58:14.954610 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:14.954629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:14.954825 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:14.955233 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:14.955240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:14.955720 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:14.955992 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.956000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.957717 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.957721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.958767 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.958778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.959787 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.959797 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.960066 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.960713 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:14.960743 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:14.962061 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:14.965737 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:14.965742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:14.965746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:14.966367 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:14.966820 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:14.966825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:14.966829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:14.966961 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:14.967472 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:14.967477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:14.967674 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 14:58:14.967859 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 14:58:14.968220 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":851,"threads_started":1}
I20260925 14:58:14.970990 6270 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:45209
I20260925 14:58:14.970963 6404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:45209 every 8 connection(s)
I20260925 14:58:14.972399 6270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:14.974316 6405 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44487
I20260925 14:58:14.974509 6405 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:14.975212 6405 heartbeater.cc:511] Master 127.4.243.254:44487 requested a full tablet report, sending...
I20260925 14:58:14.977016 6199 ts_manager.cc:194] Registered new tserver with Master: e91f97e8f533437a8fe1ededb7891df1 (127.4.243.193:45209)
I20260925 14:58:14.978379 6199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:59521
I20260925 14:58:14.980450 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 6270
I20260925 14:58:14.980665 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:14.981328 5071 external_mini_cluster.cc:1428] Setting key c9d682d249baae3028a81a1eb3e66e95
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:15.018059 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44487
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:15.320856 6409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:15.321123 6409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:15.321302 6409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:15.328258 6409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:15.328389 6409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:15.328531 6409 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:15.328685 6409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:15.338198 6409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:44487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:15.339476 6409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:15.340749 6409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:15.353279 6414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:15.353313 6417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:15.353348 6415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:15.354137 6409 server_base.cc:1034] running on GCE node
I20260925 14:58:15.354652 6409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:15.355444 6409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:15.356652 6409 hybrid_clock.cc:648] HybridClock initialized: now 1790348295356578 us; error 57 us; skew 500 ppm
I20260925 14:58:15.356956 6409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:15 dist-test-slave-qxh0 krb5kdc[6154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:15.363205 6409 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:15.364471 6409 webserver.cc:460] Webserver started at http://127.4.243.194:40655/ using document root <none> and password file <none>
I20260925 14:58:15.365084 6409 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:15.365165 6409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:15.365409 6409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:15.368257 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "308a1a32bc9b4d289dc9cfc17bc77554"
format_stamp: "Formatted at 2026-09-25 14:58:15 on dist-test-slave-qxh0"
server_key: "3a673b054b38eadda1a70dea7e578b09"
server_key_iv: "de6a79fb51eff86f347aa0b5c16ed96d"
server_key_version: "encryptionkey@0"
I20260925 14:58:15.368935 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "308a1a32bc9b4d289dc9cfc17bc77554"
format_stamp: "Formatted at 2026-09-25 14:58:15 on dist-test-slave-qxh0"
server_key: "3a673b054b38eadda1a70dea7e578b09"
server_key_iv: "de6a79fb51eff86f347aa0b5c16ed96d"
server_key_version: "encryptionkey@0"
I20260925 14:58:15.373677 6409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 14:58:15.376888 6424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:15.378188 6409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:58:15.378356 6409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "308a1a32bc9b4d289dc9cfc17bc77554"
format_stamp: "Formatted at 2026-09-25 14:58:15 on dist-test-slave-qxh0"
server_key: "3a673b054b38eadda1a70dea7e578b09"
server_key_iv: "de6a79fb51eff86f347aa0b5c16ed96d"
server_key_version: "encryptionkey@0"
I20260925 14:58:15.378566 6409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:15.404224 6409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:15.408231 6409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:15.408488 6409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:15.410887 6409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:15.410986 6409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:15.411140 6409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:15.411234 6409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:15 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:15.438386 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.411307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:46979 (local address 127.4.243.254:44487)
0925 14:58:15.411499 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:15.411508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:15.412927 (+ 1419us) server_negotiation.cc:407] Connection header received
0925 14:58:15.415861 (+ 2934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:15.415867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:15.415963 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:15.416106 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:15.418679 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.419650 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.422900 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.423676 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.423825 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.430488 (+ 6663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:15.430496 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:15.430502 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:15.430548 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:15.434471 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.435410 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.435420 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.435426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.435589 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.436262 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.436270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.436276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.436706 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:15.436865 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:15.437618 (+ 753us) server_negotiation.cc:299] Negotiation successful
0925 14:58:15.437909 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:58:15.439062 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.411954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44487 (local address 127.4.243.194:46979)
0925 14:58:15.412748 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:15.412798 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:15.415524 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:15.416397 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:15.416412 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:15.417417 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:15.418421 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.418451 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.419857 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.419863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.422646 (+ 2783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.422666 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.424065 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.424088 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.424370 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.425281 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:15.425313 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:15.430187 (+ 4874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:15.434687 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.434698 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.434722 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.435256 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.435740 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.435747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.435753 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.436100 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.436858 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:15.436869 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:15.437208 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 14:58:15.437493 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 14:58:15.437860 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":303,"threads_started":1}
I20260925 14:58:15.458168 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.445291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:34215 (local address 127.4.243.254:44487)
0925 14:58:15.445489 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:15.445497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:15.445839 (+ 342us) server_negotiation.cc:407] Connection header received
0925 14:58:15.446015 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:15.446024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:15.446115 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:15.446259 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:15.447603 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.448307 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.449611 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.450025 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.450171 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.451818 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:15.451825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:15.451830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:15.451865 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:15.455255 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.456176 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.456180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.456182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.456287 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.456639 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.456645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.456649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.457045 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:15.457198 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:15.457577 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 14:58:15.457806 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:15.458262 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.444821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44487 (local address 127.4.243.194:34215)
0925 14:58:15.445729 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:15.445762 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:15.445900 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:15.446273 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:15.446276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:15.446933 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:15.447418 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.447440 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.448445 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.448451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.449462 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.449472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.450213 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.450219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.450374 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.450985 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:15.451001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:15.451646 (+ 645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:15.455442 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.455448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.455454 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.456012 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.456398 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.456403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.456407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.456530 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.457168 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:15.457172 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:15.457351 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:58:15.457493 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 14:58:15.457806 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":675,"threads_started":1}
I20260925 14:58:15.464489 6409 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:35873
I20260925 14:58:15.464511 6543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:35873 every 8 connection(s)
I20260925 14:58:15.465642 6409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:15.467483 6544 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44487
I20260925 14:58:15.467653 6544 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:15.468187 6544 heartbeater.cc:511] Master 127.4.243.254:44487 requested a full tablet report, sending...
I20260925 14:58:15.469388 6199 ts_manager.cc:194] Registered new tserver with Master: 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873)
I20260925 14:58:15.470222 6199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:46979
I20260925 14:58:15.473078 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 6409
I20260925 14:58:15.473292 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:15.473739 5071 external_mini_cluster.cc:1428] Setting key 104d112f6112c0f78b8d27c0547da123
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:15.511904 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44487
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:15.813457 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:15.813704 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:15.813859 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:15.820938 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:15.821028 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:15.821152 6548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:15.821256 6548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:15.830360 6548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:44487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:15.831475 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:15.832743 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:15.845175 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:15.845152 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:15.845269 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:15.851246 6548 server_base.cc:1034] running on GCE node
I20260925 14:58:15.851781 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:15.852672 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:15.853906 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790348295853832 us; error 51 us; skew 500 ppm
I20260925 14:58:15.854189 6548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:15 dist-test-slave-qxh0 krb5kdc[6154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:15.857206 6548 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:15.858667 6548 webserver.cc:460] Webserver started at http://127.4.243.195:39267/ using document root <none> and password file <none>
I20260925 14:58:15.859305 6548 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:15.859385 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:15.859619 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:15.862201 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "76c6c0a00bee40a585c7475a678f4b6d"
format_stamp: "Formatted at 2026-09-25 14:58:15 on dist-test-slave-qxh0"
server_key: "5a3e4181ad5278fe331dbb8d2f765084"
server_key_iv: "65803db642838bacc4d08d18aa08bed1"
server_key_version: "encryptionkey@0"
I20260925 14:58:15.862955 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "76c6c0a00bee40a585c7475a678f4b6d"
format_stamp: "Formatted at 2026-09-25 14:58:15 on dist-test-slave-qxh0"
server_key: "5a3e4181ad5278fe331dbb8d2f765084"
server_key_iv: "65803db642838bacc4d08d18aa08bed1"
server_key_version: "encryptionkey@0"
I20260925 14:58:15.867540 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:58:15.870684 6563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:15.871882 6548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:58:15.871997 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "76c6c0a00bee40a585c7475a678f4b6d"
format_stamp: "Formatted at 2026-09-25 14:58:15 on dist-test-slave-qxh0"
server_key: "5a3e4181ad5278fe331dbb8d2f765084"
server_key_iv: "65803db642838bacc4d08d18aa08bed1"
server_key_version: "encryptionkey@0"
I20260925 14:58:15.872164 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:15.885144 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:15.888609 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:15.888815 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:15.891120 6548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:15.891192 6548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:15.891337 6548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:15.891429 6548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:15 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:15.915508 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.891648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:60197 (local address 127.4.243.254:44487)
0925 14:58:15.891828 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:15.891835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:15.893170 (+ 1335us) server_negotiation.cc:407] Connection header received
0925 14:58:15.895265 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:15.895270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:15.895413 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:15.895522 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:15.898393 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.899457 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.902435 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.902925 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.903067 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.908072 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:15.908078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:15.908081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:15.908110 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:15.911505 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.912631 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.912639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.912645 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.912810 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.913362 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.913369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.913374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.913668 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:15.913815 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:15.914761 (+ 946us) server_negotiation.cc:299] Negotiation successful
0925 14:58:15.915028 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:58:15.916608 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.892166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44487 (local address 127.4.243.195:60197)
0925 14:58:15.892987 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:15.893031 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:15.894935 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:15.895925 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:15.895941 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:15.897039 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:15.898168 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.898195 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.899629 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.899638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.902268 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.902295 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.903195 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.903202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.903629 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.904385 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:15.904426 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:15.907848 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:15.911672 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.911679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.911697 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.912432 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.912944 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.912947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.912949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.913182 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.913812 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:15.913820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:15.914327 (+ 507us) client_negotiation.cc:769] Sending connection context
0925 14:58:15.914652 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 14:58:15.915253 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":350,"threads_started":1}
I20260925 14:58:15.946273 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.929922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:36839 (local address 127.4.243.254:44487)
0925 14:58:15.930101 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:15.930108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:15.930833 (+ 725us) server_negotiation.cc:407] Connection header received
0925 14:58:15.931009 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:15.931017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:15.931111 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:15.931223 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:15.932572 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.933603 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.935981 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.936656 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.936766 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.939938 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:15.939944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:15.939949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:15.939985 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:15.943191 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.944043 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.944050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.944054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.944245 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:15.944691 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:15.944695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:15.944699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:15.945038 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:15.945213 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:15.945682 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 14:58:15.945884 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:58:15.946391 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.930004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44487 (local address 127.4.243.195:36839)
0925 14:58:15.930715 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:15.930736 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:15.930870 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:15.931264 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:15.931268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:15.931837 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:15.932265 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.932282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.933944 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.933949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.935783 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:15.935800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.937065 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.937077 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:15.937309 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.938147 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:15.938172 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:15.939751 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:15.943311 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.943316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.943324 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.943916 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.944387 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:15.944397 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:15.944401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:15.944578 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:15.945151 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:15.945155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:15.945390 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 14:58:15.945570 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 14:58:15.945898 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":355,"threads_started":1}
I20260925 14:58:15.960953 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:41123 every 8 connection(s)
I20260925 14:58:15.960994 6548 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:41123
I20260925 14:58:15.962226 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:15.964236 6683 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44487
I20260925 14:58:15.964478 6683 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:15.965077 6683 heartbeater.cc:511] Master 127.4.243.254:44487 requested a full tablet report, sending...
I20260925 14:58:15.966354 6199 ts_manager.cc:194] Registered new tserver with Master: 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195:41123)
I20260925 14:58:15.967201 6199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:60197
I20260925 14:58:15.969172 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 6548
I20260925 14:58:15.969357 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:15.969792 5071 external_mini_cluster.cc:1428] Setting key 70146bab877852d4193791a7055c7aae
I20260925 14:58:15.972620 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:15.981140 6405 heartbeater.cc:503] Master 127.4.243.254:44487 was elected leader, sending a full tablet report...
Sep 25 14:58:15 dist-test-slave-qxh0 krb5kdc[6154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:16 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:16.015872 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.003004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.4.243.254:44487)
0925 14:58:16.003157 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.003161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.003511 (+ 350us) server_negotiation.cc:407] Connection header received
0925 14:58:16.003698 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.003701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.003778 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.003908 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:16.005158 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.005871 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.007123 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.007506 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.007597 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.011128 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:16.011133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:16.011136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:16.011167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:16.013573 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:16.014224 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:16.014228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:16.014230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:16.014335 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:16.014707 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:16.014710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:16.014712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:16.014952 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:16.015119 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.015383 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.015547 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:58:16.019598 6199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:16.025801 6199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:16.054525 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.037095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.4.243.193:45209)
0925 14:58:16.046264 (+ 9169us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.046275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.046308 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:58:16.046453 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.046463 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.046769 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.047153 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:16.048113 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.049442 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.052005 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.053087 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.053228 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.053443 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.054085 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.054251 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8995,"thread_start_us":8844,"threads_started":1}
I20260925 14:58:16.054507 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.037116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41123 (local address 127.0.0.1:55342)
0925 14:58:16.045572 (+ 8456us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.045592 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:16.045702 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:16.046980 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:16.046987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:16.047014 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:16.047416 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.047429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.049169 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.049177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.051671 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.051685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.053616 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.053621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.053815 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.053851 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:58:16.053936 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:58:16.054058 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8312,"mutex_wait_us":179,"thread_start_us":8177,"threads_started":1}
I20260925 14:58:16.055099 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.036697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:45209 (local address 127.0.0.1:46692)
0925 14:58:16.044889 (+ 8192us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.044909 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:16.045036 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:16.047011 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:16.047018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:16.047054 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:16.047943 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.047957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.049570 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.049575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.051880 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.051889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.053265 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.053270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.053454 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.053627 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 14:58:16.053753 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 14:58:16.054728 (+ 975us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8045,"mutex_wait_us":556,"thread_start_us":7894,"threads_started":1}
I20260925 14:58:16.055291 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.037041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.4.243.194:35873)
0925 14:58:16.047716 (+ 10675us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.047723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.047756 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:58:16.047822 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.047826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.047970 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.048143 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:16.048652 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.049850 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.052062 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.053321 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.053448 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.053644 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.054610 (+ 966us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.054998 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10519,"thread_start_us":10394,"threads_started":1}
I20260925 14:58:16.055431 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.037712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.4.243.195:41123)
0925 14:58:16.045996 (+ 8284us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.046006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.046039 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:58:16.046130 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.046138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.046413 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.046658 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:16.047609 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.049052 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.051877 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.053333 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.053448 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.053661 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.054049 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.055090 (+ 1041us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8079,"thread_start_us":7892,"threads_started":1}
I20260925 14:58:16.055724 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.036429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:35873 (local address 127.0.0.1:37658)
0925 14:58:16.043365 (+ 6936us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.043405 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:16.043553 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:16.048135 (+ 4582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:16.048140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:16.048164 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:16.048468 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.048477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.050045 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.050050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.051898 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.051910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.053817 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.053822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.054039 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.054072 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:58:16.054162 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:58:16.054287 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6718,"mutex_wait_us":43,"thread_start_us":6604,"threads_started":1}
I20260925 14:58:16.058306 6335 tablet_service.cc:1467] Processing CreateTablet for tablet fb56c305301c465a856e323d92c27e9c (DEFAULT_TABLE table=test-table [id=f723426193964ff496d5333d5b6cdd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:16.058634 6613 tablet_service.cc:1467] Processing CreateTablet for tablet fb56c305301c465a856e323d92c27e9c (DEFAULT_TABLE table=test-table [id=f723426193964ff496d5333d5b6cdd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:16.058863 6474 tablet_service.cc:1467] Processing CreateTablet for tablet fb56c305301c465a856e323d92c27e9c (DEFAULT_TABLE table=test-table [id=f723426193964ff496d5333d5b6cdd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:16.060001 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb56c305301c465a856e323d92c27e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.059995 6335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb56c305301c465a856e323d92c27e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.059994 6613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb56c305301c465a856e323d92c27e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.070266 6702 tablet_bootstrap.cc:492] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Bootstrap starting.
I20260925 14:58:16.073614 6702 tablet_bootstrap.cc:654] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.074751 6702 log.cc:824] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:16.079375 6703 tablet_bootstrap.cc:492] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Bootstrap starting.
I20260925 14:58:16.080364 6704 tablet_bootstrap.cc:492] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Bootstrap starting.
I20260925 14:58:16.085585 6704 tablet_bootstrap.cc:654] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.086724 6704 log.cc:824] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:16.088411 6702 tablet_bootstrap.cc:492] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: No bootstrap required, opened a new log
I20260925 14:58:16.088670 6702 ts_tablet_manager.cc:1397] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260925 14:58:16.088699 6703 tablet_bootstrap.cc:654] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.089624 6704 tablet_bootstrap.cc:492] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: No bootstrap required, opened a new log
I20260925 14:58:16.089646 6703 log.cc:824] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:16.089860 6704 ts_tablet_manager.cc:1397] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 14:58:16.092377 6703 tablet_bootstrap.cc:492] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: No bootstrap required, opened a new log
I20260925 14:58:16.092617 6703 ts_tablet_manager.cc:1397] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260925 14:58:16.099278 6702 raft_consensus.cc:348] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.099750 6702 raft_consensus.cc:374] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.099851 6702 raft_consensus.cc:729] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91f97e8f533437a8fe1ededb7891df1, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.099915 6703 raft_consensus.cc:348] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.100369 6703 raft_consensus.cc:374] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.100467 6703 raft_consensus.cc:729] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308a1a32bc9b4d289dc9cfc17bc77554, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.100266 6704 raft_consensus.cc:348] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.100654 6704 raft_consensus.cc:374] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.100487 6702 consensus_queue.cc:260] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.100741 6704 raft_consensus.cc:729] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c6c0a00bee40a585c7475a678f4b6d, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.101039 6703 consensus_queue.cc:260] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.101399 6704 consensus_queue.cc:260] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.102345 6702 ts_tablet_manager.cc:1428] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260925 14:58:16.103500 6544 heartbeater.cc:503] Master 127.4.243.254:44487 was elected leader, sending a full tablet report...
I20260925 14:58:16.104072 6683 heartbeater.cc:503] Master 127.4.243.254:44487 was elected leader, sending a full tablet report...
I20260925 14:58:16.104598 6704 ts_tablet_manager.cc:1428] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260925 14:58:16.106067 6703 ts_tablet_manager.cc:1428] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
W20260925 14:58:16.213935 6684 tablet.cc:2369] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:16.217589 6545 tablet.cc:2369] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:16.224843 6406 tablet.cc:2369] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:16.372364 6708 raft_consensus.cc:482] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:16.372679 6708 raft_consensus.cc:504] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.374372 6708 leader_election.cc:288] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873), 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195:41123)
I20260925 14:58:16.381505 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.374838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:50151 (local address 127.4.243.194:35873)
0925 14:58:16.375184 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.375191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.375956 (+ 765us) server_negotiation.cc:407] Connection header received
0925 14:58:16.376046 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.376052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.376128 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.376231 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:16.376843 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.377747 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.379731 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.380570 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.380687 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.380741 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.381100 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.381236 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260925 14:58:16.381632 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.374661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:35873 (local address 127.4.243.193:50151)
0925 14:58:16.375475 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.375495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:16.375663 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:16.376259 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:16.376262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:16.376284 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:16.376685 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.376697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.377866 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.377872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.379582 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.379593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.380757 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.380764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.380921 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.380952 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 14:58:16.381080 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 14:58:16.381238 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"spinlock_wait_cycles":35712,"thread_start_us":394,"threads_started":1}
I20260925 14:58:16.381897 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.375370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:35659 (local address 127.4.243.195:41123)
0925 14:58:16.375558 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.375565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.376328 (+ 763us) server_negotiation.cc:407] Connection header received
0925 14:58:16.376643 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.376649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.376751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.376842 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:16.377372 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.378261 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.380094 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.380936 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.381099 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.381161 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.381491 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.381634 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:58:16.382187 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.375000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41123 (local address 127.4.243.193:35659)
0925 14:58:16.376245 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.376263 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:16.376487 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:16.376858 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:16.376863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:16.376887 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:16.377234 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.377244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.378447 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.378452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.379955 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.379968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.381147 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.381153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.381327 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.381363 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:58:16.381454 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:58:16.381592 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"thread_start_us":886,"threads_started":1}
I20260925 14:58:16.382970 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb56c305301c465a856e323d92c27e9c" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" is_pre_election: true
I20260925 14:58:16.383237 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb56c305301c465a856e323d92c27e9c" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c6c0a00bee40a585c7475a678f4b6d" is_pre_election: true
I20260925 14:58:16.383555 6494 raft_consensus.cc:2454] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 0.
I20260925 14:58:16.383659 6638 raft_consensus.cc:2454] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 0.
I20260925 14:58:16.384213 6286 leader_election.cc:302] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308a1a32bc9b4d289dc9cfc17bc77554, e91f97e8f533437a8fe1ededb7891df1; no voters:
I20260925 14:58:16.384537 6708 raft_consensus.cc:2781] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:16.384675 6708 raft_consensus.cc:482] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:16.384783 6708 raft_consensus.cc:3037] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:16.387449 6708 raft_consensus.cc:504] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.388003 6708 leader_election.cc:288] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 election: Requested vote from peers 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873), 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195:41123)
I20260925 14:58:16.388356 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb56c305301c465a856e323d92c27e9c" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308a1a32bc9b4d289dc9cfc17bc77554"
I20260925 14:58:16.388474 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb56c305301c465a856e323d92c27e9c" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c6c0a00bee40a585c7475a678f4b6d"
I20260925 14:58:16.388546 6494 raft_consensus.cc:3037] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:16.388692 6638 raft_consensus.cc:3037] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:16.390513 6494 raft_consensus.cc:2454] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 1.
I20260925 14:58:16.390772 6638 raft_consensus.cc:2454] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 1.
I20260925 14:58:16.390918 6286 leader_election.cc:302] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308a1a32bc9b4d289dc9cfc17bc77554, e91f97e8f533437a8fe1ededb7891df1; no voters:
I20260925 14:58:16.391165 6708 raft_consensus.cc:2781] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:16.391512 6708 raft_consensus.cc:686] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 1 LEADER]: Becoming Leader. State: Replica: e91f97e8f533437a8fe1ededb7891df1, State: Running, Role: LEADER
I20260925 14:58:16.391944 6708 consensus_queue.cc:237] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.396222 6199 catalog_manager.cc:5495] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e91f97e8f533437a8fe1ededb7891df1 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "e91f97e8f533437a8fe1ededb7891df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:16.422305 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.417166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.4.243.193:45209)
0925 14:58:16.417377 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.417383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.417402 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:58:16.417613 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.417616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.417704 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.417841 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:16.418173 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.418995 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.420283 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.420664 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.420793 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.421259 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:16.421698 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:16.421847 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.421986 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.422056 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:58:16.429903 6494 raft_consensus.cc:1261] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Refusing update from remote peer e91f97e8f533437a8fe1ededb7891df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:16.429903 6638 raft_consensus.cc:1261] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Refusing update from remote peer e91f97e8f533437a8fe1ededb7891df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:16.430824 6713 consensus_queue.cc:1035] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:16.431125 6708 consensus_queue.cc:1035] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:16.463109 6196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:16.463435 6196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:16.467396 6196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210: Sending DeleteTablet for 3 replicas of tablet fb56c305301c465a856e323d92c27e9c
I20260925 14:58:16.468609 6474 tablet_service.cc:1514] Processing DeleteTablet for tablet fb56c305301c465a856e323d92c27e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:16 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:37658
I20260925 14:58:16.468892 6613 tablet_service.cc:1514] Processing DeleteTablet for tablet fb56c305301c465a856e323d92c27e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:16 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:55342
I20260925 14:58:16.469774 6335 tablet_service.cc:1514] Processing DeleteTablet for tablet fb56c305301c465a856e323d92c27e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:16 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:46692
I20260925 14:58:16.477667 6196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:16.478318 6196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:16.489737 6473 tablet_service.cc:1467] Processing CreateTablet for tablet 8feb1a49db134429978387f18fe54865 (DEFAULT_TABLE table=test-table [id=7d311f4eb49e4eeab73f6b7097bfed89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:16.490427 6473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feb1a49db134429978387f18fe54865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.491320 6334 tablet_service.cc:1467] Processing CreateTablet for tablet 8feb1a49db134429978387f18fe54865 (DEFAULT_TABLE table=test-table [id=7d311f4eb49e4eeab73f6b7097bfed89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:16.491412 6612 tablet_service.cc:1467] Processing CreateTablet for tablet 8feb1a49db134429978387f18fe54865 (DEFAULT_TABLE table=test-table [id=7d311f4eb49e4eeab73f6b7097bfed89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:16.491822 6612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feb1a49db134429978387f18fe54865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.491865 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feb1a49db134429978387f18fe54865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.502490 6702 tablet_bootstrap.cc:492] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: Bootstrap starting.
I20260925 14:58:16.504532 6702 tablet_bootstrap.cc:654] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.505406 6704 tablet_bootstrap.cc:492] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: Bootstrap starting.
I20260925 14:58:16.507395 6704 tablet_bootstrap.cc:654] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.510957 6703 tablet_bootstrap.cc:492] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: Bootstrap starting.
I20260925 14:58:16.513305 6703 tablet_bootstrap.cc:654] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.515106 6727 tablet_replica.cc:330] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: stopping tablet replica
I20260925 14:58:16.515628 6727 raft_consensus.cc:2229] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:16.516160 6727 raft_consensus.cc:2258] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:16.516911 6726 tablet_replica.cc:330] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: stopping tablet replica
I20260925 14:58:16.517287 6728 tablet_replica.cc:330] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: stopping tablet replica
I20260925 14:58:16.517375 6726 raft_consensus.cc:2229] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:16.517920 6726 raft_consensus.cc:2258] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:16.518486 6728 raft_consensus.cc:2229] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:16.521348 6726 ts_tablet_manager.cc:1905] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:16.522113 6728 raft_consensus.cc:2258] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:16.524132 6727 ts_tablet_manager.cc:1905] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:16.528927 6728 ts_tablet_manager.cc:1905] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:16.532807 6704 tablet_bootstrap.cc:492] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: No bootstrap required, opened a new log
I20260925 14:58:16.532948 6704 ts_tablet_manager.cc:1397] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.003s
I20260925 14:58:16.533869 6704 raft_consensus.cc:348] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.534112 6704 raft_consensus.cc:374] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.534200 6704 raft_consensus.cc:729] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c6c0a00bee40a585c7475a678f4b6d, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.535460 6703 tablet_bootstrap.cc:492] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: No bootstrap required, opened a new log
I20260925 14:58:16.535629 6703 ts_tablet_manager.cc:1397] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.010s
I20260925 14:58:16.536361 6703 raft_consensus.cc:348] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.536581 6703 raft_consensus.cc:374] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.536675 6703 raft_consensus.cc:729] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308a1a32bc9b4d289dc9cfc17bc77554, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.537402 6703 consensus_queue.cc:260] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.540246 6703 ts_tablet_manager.cc:1428] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 14:58:16.542321 6704 consensus_queue.cc:260] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.544799 6704 ts_tablet_manager.cc:1428] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20260925 14:58:16.547576 6702 tablet_bootstrap.cc:492] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: No bootstrap required, opened a new log
I20260925 14:58:16.547757 6702 ts_tablet_manager.cc:1397] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.021s
I20260925 14:58:16.548672 6702 raft_consensus.cc:348] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.548909 6702 raft_consensus.cc:374] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.549005 6702 raft_consensus.cc:729] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91f97e8f533437a8fe1ededb7891df1, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.549405 6726 ts_tablet_manager.cc:1918] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:16.549559 6726 log.cc:1196] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/fb56c305301c465a856e323d92c27e9c
I20260925 14:58:16.549724 6702 consensus_queue.cc:260] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.550412 6702 ts_tablet_manager.cc:1428] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:16.550542 6727 ts_tablet_manager.cc:1918] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:16.550729 6727 log.cc:1196] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/fb56c305301c465a856e323d92c27e9c
I20260925 14:58:16.551220 6727 ts_tablet_manager.cc:1939] T fb56c305301c465a856e323d92c27e9c P 76c6c0a00bee40a585c7475a678f4b6d: Deleting consensus metadata
I20260925 14:58:16.551614 6726 ts_tablet_manager.cc:1939] T fb56c305301c465a856e323d92c27e9c P 308a1a32bc9b4d289dc9cfc17bc77554: Deleting consensus metadata
I20260925 14:58:16.552875 6186 catalog_manager.cc:4841] TS 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195:41123): tablet fb56c305301c465a856e323d92c27e9c (table test-table [id=f723426193964ff496d5333d5b6cdd5a]) successfully deleted
I20260925 14:58:16.554323 6728 ts_tablet_manager.cc:1918] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:16.554484 6728 log.cc:1196] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/fb56c305301c465a856e323d92c27e9c
I20260925 14:58:16.554500 6183 catalog_manager.cc:4841] TS 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873): tablet fb56c305301c465a856e323d92c27e9c (table test-table [id=f723426193964ff496d5333d5b6cdd5a]) successfully deleted
I20260925 14:58:16.555091 6728 ts_tablet_manager.cc:1939] T fb56c305301c465a856e323d92c27e9c P e91f97e8f533437a8fe1ededb7891df1: Deleting consensus metadata
I20260925 14:58:16.556301 6185 catalog_manager.cc:4841] TS e91f97e8f533437a8fe1ededb7891df1 (127.4.243.193:45209): tablet fb56c305301c465a856e323d92c27e9c (table test-table [id=f723426193964ff496d5333d5b6cdd5a]) successfully deleted
I20260925 14:58:16.837728 6708 raft_consensus.cc:482] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:16.837972 6708 raft_consensus.cc:504] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.838708 6708 leader_election.cc:288] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873), 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195:41123)
I20260925 14:58:16.839267 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feb1a49db134429978387f18fe54865" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c6c0a00bee40a585c7475a678f4b6d" is_pre_election: true
I20260925 14:58:16.839274 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feb1a49db134429978387f18fe54865" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" is_pre_election: true
I20260925 14:58:16.839524 6494 raft_consensus.cc:2454] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 0.
I20260925 14:58:16.839540 6638 raft_consensus.cc:2454] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 0.
I20260925 14:58:16.839979 6286 leader_election.cc:302] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308a1a32bc9b4d289dc9cfc17bc77554, e91f97e8f533437a8fe1ededb7891df1; no voters:
I20260925 14:58:16.840271 6708 raft_consensus.cc:2781] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:16.840847 6708 raft_consensus.cc:482] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:16.840981 6708 raft_consensus.cc:3037] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:16.843251 6708 raft_consensus.cc:504] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.843847 6708 leader_election.cc:288] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 election: Requested vote from peers 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873), 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195:41123)
I20260925 14:58:16.844264 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feb1a49db134429978387f18fe54865" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308a1a32bc9b4d289dc9cfc17bc77554"
I20260925 14:58:16.844339 6638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feb1a49db134429978387f18fe54865" candidate_uuid: "e91f97e8f533437a8fe1ededb7891df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c6c0a00bee40a585c7475a678f4b6d"
I20260925 14:58:16.844465 6494 raft_consensus.cc:3037] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:16.844542 6638 raft_consensus.cc:3037] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:16.846715 6494 raft_consensus.cc:2454] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 1.
I20260925 14:58:16.847139 6286 leader_election.cc:302] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308a1a32bc9b4d289dc9cfc17bc77554, e91f97e8f533437a8fe1ededb7891df1; no voters:
I20260925 14:58:16.847422 6708 raft_consensus.cc:2781] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:16.847421 6638 raft_consensus.cc:2454] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91f97e8f533437a8fe1ededb7891df1 in term 1.
I20260925 14:58:16.847582 6708 raft_consensus.cc:686] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 1 LEADER]: Becoming Leader. State: Replica: e91f97e8f533437a8fe1ededb7891df1, State: Running, Role: LEADER
I20260925 14:58:16.847977 6708 consensus_queue.cc:237] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.850957 6199 catalog_manager.cc:5495] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e91f97e8f533437a8fe1ededb7891df1 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "e91f97e8f533437a8fe1ededb7891df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:58:16 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348294, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:16.895323 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.880723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44487 (local address 127.0.0.1:37200)
0925 14:58:16.881549 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.881566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:16.881716 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:16.882056 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:16.882061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:16.883015 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:16.883178 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.883187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.884566 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.884570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.885901 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:16.885912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.886861 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.886865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:16.886986 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.887642 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:16.887677 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:16.889589 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:16.892902 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:16.892910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:16.892915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:16.893289 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:16.893626 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:16.893629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:16.893631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:16.893765 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:16.894320 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:16.894326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:16.894482 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 14:58:16.894721 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 14:58:16.894976 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":565,"threads_started":1}
I20260925 14:58:16.895807 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.880723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.4.243.254:44487)
0925 14:58:16.881247 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:16.881253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:16.881675 (+ 422us) server_negotiation.cc:407] Connection header received
0925 14:58:16.881815 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:16.881820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:16.881892 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:16.881995 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:16.883305 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.884419 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.886043 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:16.886700 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:16.886822 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:16.889763 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:16.889768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:16.889773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:16.889804 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:16.892752 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:16.893395 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:16.893400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:16.893403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:16.893529 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:16.893860 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:16.893864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:16.893868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:16.894175 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:16.894308 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:16.894804 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 14:58:16.895059 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":384,"thread_start_us":261,"threads_started":1}
I20260925 14:58:16.900454 6199 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:37200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:58:16.901446 6199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:16.907373 6335 tablet_service.cc:1467] Processing CreateTablet for tablet 6b77515e563844b7bb5f24efbdecccac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=329114c5534848ec8bfd28048a0cf6f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:16.907619 6473 tablet_service.cc:1467] Processing CreateTablet for tablet 6b77515e563844b7bb5f24efbdecccac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=329114c5534848ec8bfd28048a0cf6f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:16.907939 6335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b77515e563844b7bb5f24efbdecccac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.907862 6613 tablet_service.cc:1467] Processing CreateTablet for tablet 6b77515e563844b7bb5f24efbdecccac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=329114c5534848ec8bfd28048a0cf6f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:16.908120 6473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b77515e563844b7bb5f24efbdecccac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.908370 6613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b77515e563844b7bb5f24efbdecccac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:16.913098 6702 tablet_bootstrap.cc:492] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: Bootstrap starting.
I20260925 14:58:16.914420 6702 tablet_bootstrap.cc:654] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.914958 6704 tablet_bootstrap.cc:492] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: Bootstrap starting.
I20260925 14:58:16.915721 6703 tablet_bootstrap.cc:492] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: Bootstrap starting.
I20260925 14:58:16.916266 6702 tablet_bootstrap.cc:492] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: No bootstrap required, opened a new log
I20260925 14:58:16.916436 6702 ts_tablet_manager.cc:1397] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:16.916486 6704 tablet_bootstrap.cc:654] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.917065 6702 raft_consensus.cc:348] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.917313 6702 raft_consensus.cc:374] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.917382 6702 raft_consensus.cc:729] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91f97e8f533437a8fe1ededb7891df1, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.917693 6703 tablet_bootstrap.cc:654] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:16.917682 6702 consensus_queue.cc:260] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.918087 6708 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:16.919333 6708 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:16.919747 6702 ts_tablet_manager.cc:1428] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:16.919932 6703 tablet_bootstrap.cc:492] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: No bootstrap required, opened a new log
I20260925 14:58:16.919942 6704 tablet_bootstrap.cc:492] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: No bootstrap required, opened a new log
I20260925 14:58:16.920066 6703 ts_tablet_manager.cc:1397] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 14:58:16.920104 6704 ts_tablet_manager.cc:1397] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 14:58:16.920717 6704 raft_consensus.cc:348] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.920718 6703 raft_consensus.cc:348] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.920861 6704 raft_consensus.cc:374] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.920904 6704 raft_consensus.cc:729] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c6c0a00bee40a585c7475a678f4b6d, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.920920 6703 raft_consensus.cc:374] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:16.921006 6703 raft_consensus.cc:729] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308a1a32bc9b4d289dc9cfc17bc77554, State: Initialized, Role: FOLLOWER
I20260925 14:58:16.921096 6704 consensus_queue.cc:260] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.921234 6703 consensus_queue.cc:260] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:16.921603 6704 ts_tablet_manager.cc:1428] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:16.921686 6703 ts_tablet_manager.cc:1428] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:58:16.921516 6710 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:16.921859 6710 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:16.922114 6709 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:16.922417 6709 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:17.180279 6710 raft_consensus.cc:482] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:17.180454 6710 raft_consensus.cc:504] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:17.182017 6710 leader_election.cc:288] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e91f97e8f533437a8fe1ededb7891df1 (127.4.243.193:45209), 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873)
I20260925 14:58:17.188194 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.182315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:56197 (local address 127.4.243.193:45209)
0925 14:58:17.183385 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.183390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.183405 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:58:17.183467 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.183470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.183524 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.183644 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:17.184129 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.184800 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.186488 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.187290 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.187463 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.187538 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.187812 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.187934 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":876,"threads_started":1}
I20260925 14:58:17.188231 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.182199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:45209 (local address 127.4.243.195:56197)
0925 14:58:17.182740 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:17.182765 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:17.182903 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:17.183660 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:17.183664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:17.183687 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:17.183968 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.183977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.184951 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.184956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.186303 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.186314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.187443 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.187449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.187667 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.187689 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:58:17.187789 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:58:17.187913 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":268,"threads_started":1}
I20260925 14:58:17.188271 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.182532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:47879 (local address 127.4.243.194:35873)
0925 14:58:17.183042 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.183051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.183475 (+ 424us) server_negotiation.cc:407] Connection header received
0925 14:58:17.183628 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.183632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.183685 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.183788 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:17.184384 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.184991 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.186614 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.187335 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.187509 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.187580 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.187857 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.187976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260925 14:58:17.188463 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.182269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:35873 (local address 127.4.243.195:47879)
0925 14:58:17.183377 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:17.183389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:17.183508 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:17.183802 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:17.183806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:17.183847 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:17.184216 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.184224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.185126 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.185133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.186445 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.186456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.187533 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.187539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.187714 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.187734 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 14:58:17.187833 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:58:17.187949 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":869,"threads_started":1}
I20260925 14:58:17.188997 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b77515e563844b7bb5f24efbdecccac" candidate_uuid: "76c6c0a00bee40a585c7475a678f4b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" is_pre_election: true
I20260925 14:58:17.189002 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b77515e563844b7bb5f24efbdecccac" candidate_uuid: "76c6c0a00bee40a585c7475a678f4b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91f97e8f533437a8fe1ededb7891df1" is_pre_election: true
I20260925 14:58:17.189249 6494 raft_consensus.cc:2454] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c6c0a00bee40a585c7475a678f4b6d in term 0.
I20260925 14:58:17.189330 6359 raft_consensus.cc:2454] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c6c0a00bee40a585c7475a678f4b6d in term 0.
I20260925 14:58:17.189692 6564 leader_election.cc:302] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308a1a32bc9b4d289dc9cfc17bc77554, 76c6c0a00bee40a585c7475a678f4b6d; no voters:
I20260925 14:58:17.190059 6710 raft_consensus.cc:2781] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:17.190203 6710 raft_consensus.cc:482] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:17.190290 6710 raft_consensus.cc:3037] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:17.192348 6710 raft_consensus.cc:504] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:17.192842 6710 leader_election.cc:288] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [CANDIDATE]: Term 1 election: Requested vote from peers e91f97e8f533437a8fe1ededb7891df1 (127.4.243.193:45209), 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873)
I20260925 14:58:17.193302 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b77515e563844b7bb5f24efbdecccac" candidate_uuid: "76c6c0a00bee40a585c7475a678f4b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91f97e8f533437a8fe1ededb7891df1"
I20260925 14:58:17.193287 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b77515e563844b7bb5f24efbdecccac" candidate_uuid: "76c6c0a00bee40a585c7475a678f4b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308a1a32bc9b4d289dc9cfc17bc77554"
I20260925 14:58:17.193476 6359 raft_consensus.cc:3037] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:17.193526 6494 raft_consensus.cc:3037] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:17.196316 6494 raft_consensus.cc:2454] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c6c0a00bee40a585c7475a678f4b6d in term 1.
I20260925 14:58:17.196388 6359 raft_consensus.cc:2454] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c6c0a00bee40a585c7475a678f4b6d in term 1.
I20260925 14:58:17.196720 6564 leader_election.cc:302] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 308a1a32bc9b4d289dc9cfc17bc77554, 76c6c0a00bee40a585c7475a678f4b6d, e91f97e8f533437a8fe1ededb7891df1; no voters:
I20260925 14:58:17.197042 6710 raft_consensus.cc:2781] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:17.197381 6710 raft_consensus.cc:686] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [term 1 LEADER]: Becoming Leader. State: Replica: 76c6c0a00bee40a585c7475a678f4b6d, State: Running, Role: LEADER
I20260925 14:58:17.197733 6710 consensus_queue.cc:237] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } }
I20260925 14:58:17.199532 6747 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: TxnStatusTablet state changed. Reason: New leader 76c6c0a00bee40a585c7475a678f4b6d. Latest consensus state: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:17.199906 6747 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:17.200562 6749 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:58:17.200762 6749 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:58:17.201157 6196 catalog_manager.cc:5495] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195). New cstate: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:17.219601 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.213218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.4.243.195:41123)
0925 14:58:17.213838 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.213842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.213872 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:58:17.213950 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.213953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.214015 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.214139 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:17.214584 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.215426 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.217279 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.217727 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.217814 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.218304 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:17.218997 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:17.219100 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.219238 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.219296 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":382,"threads_started":1}
I20260925 14:58:17.219717 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.213123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41123 (local address 127.0.0.1:55346)
0925 14:58:17.213372 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:17.213393 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:17.213540 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:17.214192 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:17.214197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:17.214211 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:17.214405 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.214416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.215555 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.215561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.217143 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.217158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.217866 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.217872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.218052 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.218115 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:17.219101 (+ 986us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:17.219121 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 14:58:17.219214 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:58:17.219358 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 14:58:17.223584 6359 raft_consensus.cc:1261] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Refusing update from remote peer 76c6c0a00bee40a585c7475a678f4b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:17.224264 6494 raft_consensus.cc:1261] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Refusing update from remote peer 76c6c0a00bee40a585c7475a678f4b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:17.224804 6747 consensus_queue.cc:1035] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:17.225095 6710 consensus_queue.cc:1035] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:17.228648 6708 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: TxnStatusTablet state changed. Reason: New leader 76c6c0a00bee40a585c7475a678f4b6d. Latest consensus state: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:17.228904 6708 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:17.232271 6755 mvcc.cc:205] Tried to move back new op lower bound from 7333266625420873728 to 7333266625325182976. Current Snapshot: MvccSnapshot[applied={T|T < 7333266625420873728}]
I20260925 14:58:17.236771 6753 mvcc.cc:205] Tried to move back new op lower bound from 7333266625420873728 to 7333266625325182976. Current Snapshot: MvccSnapshot[applied={T|T < 7333266625420873728}]
I20260925 14:58:17.238119 6710 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:17.238483 6710 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:17.243193 6708 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:17.243414 6708 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P e91f97e8f533437a8fe1ededb7891df1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:17.255461 6709 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: TxnStatusTablet state changed. Reason: New leader 76c6c0a00bee40a585c7475a678f4b6d. Latest consensus state: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:17.255729 6709 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:17.257052 6710 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:17.257305 6710 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P 76c6c0a00bee40a585c7475a678f4b6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:17.262830 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.253470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:52089 (local address 127.4.243.195:41123)
0925 14:58:17.253812 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.253818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.254196 (+ 378us) server_negotiation.cc:407] Connection header received
0925 14:58:17.254394 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.254399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.254492 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.254580 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:17.255231 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.256317 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.260708 (+ 4391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.261375 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.261476 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.262057 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:17.262173 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:17.262231 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.262456 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.262528 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260925 14:58:17.263756 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.253485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41123 (local address 127.4.243.193:52089)
0925 14:58:17.254071 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:17.254090 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:17.254262 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:17.254758 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:17.254761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:17.254770 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:17.255058 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.255072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.258323 (+ 3251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.258329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.260551 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.260566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.261645 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.261651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.261805 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.261867 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:17.262306 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:17.262316 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:58:17.263181 (+ 865us) client_negotiation.cc:240] Negotiation successful
0925 14:58:17.263353 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":371,"threads_started":1}
I20260925 14:58:17.267045 6709 tablet_replica.cc:439] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76c6c0a00bee40a585c7475a678f4b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91f97e8f533437a8fe1ededb7891df1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 45209 } } peers { permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 } } peers { permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 } } }
I20260925 14:58:17.267287 6709 tablet_replica.cc:442] T 6b77515e563844b7bb5f24efbdecccac P 308a1a32bc9b4d289dc9cfc17bc77554: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:17.272982 6494 raft_consensus.cc:1261] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Refusing update from remote peer e91f97e8f533437a8fe1ededb7891df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:17.273325 6638 raft_consensus.cc:1261] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Refusing update from remote peer e91f97e8f533437a8fe1ededb7891df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:17.273432 6713 consensus_queue.cc:1035] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "308a1a32bc9b4d289dc9cfc17bc77554" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:17.273849 6708 consensus_queue.cc:1035] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c6c0a00bee40a585c7475a678f4b6d" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:17.278816 6754 mvcc.cc:205] Tried to move back new op lower bound from 7333266625624444928 to 7333266623891382272. Current Snapshot: MvccSnapshot[applied={T|T < 7333266625624444928}]
I20260925 14:58:17.305579 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.298510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:57213 (local address 127.4.243.193:45209)
0925 14:58:17.298681 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.298688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.299205 (+ 517us) server_negotiation.cc:407] Connection header received
0925 14:58:17.299326 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.299331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.299415 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.299534 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:17.300187 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.301257 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.303173 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.303784 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.303900 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.304845 (+ 945us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:17.304974 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:17.305062 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.305210 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.305284 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:58:17.305675 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.298444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:45209 (local address 127.4.243.195:57213)
0925 14:58:17.299062 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:17.299089 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:17.299216 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:17.299608 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:17.299612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:17.299625 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:17.300035 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.300050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.301444 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.301450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.303043 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:17.303055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.303948 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.303953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:17.304603 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.304678 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:17.305079 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:17.305090 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:58:17.305175 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:58:17.305332 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":329,"threads_started":1}
I20260925 14:58:17.337309 6199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:17.337574 6199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:17.340973 6199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fb64934ec954b689be2be362e968210: Sending DeleteTablet for 3 replicas of tablet 8feb1a49db134429978387f18fe54865
I20260925 14:58:17.341674 6335 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feb1a49db134429978387f18fe54865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:17 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:46692
I20260925 14:58:17.341786 6473 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feb1a49db134429978387f18fe54865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:17 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:37658
I20260925 14:58:17.341774 6613 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feb1a49db134429978387f18fe54865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:17 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:55342
I20260925 14:58:17.351516 6771 tablet_replica.cc:330] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: stopping tablet replica
I20260925 14:58:17.351824 6773 tablet_replica.cc:330] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: stopping tablet replica
I20260925 14:58:17.351965 6771 raft_consensus.cc:2229] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:17.352029 6773 raft_consensus.cc:2229] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:17.352308 6773 raft_consensus.cc:2258] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:17.352553 6771 raft_consensus.cc:2258] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:17.353585 6773 ts_tablet_manager.cc:1905] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:17.353948 6771 ts_tablet_manager.cc:1905] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:17.361315 6773 ts_tablet_manager.cc:1918] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:17.361459 6773 log.cc:1196] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/8feb1a49db134429978387f18fe54865
I20260925 14:58:17.361900 6773 ts_tablet_manager.cc:1939] T 8feb1a49db134429978387f18fe54865 P 76c6c0a00bee40a585c7475a678f4b6d: Deleting consensus metadata
I20260925 14:58:17.362295 6771 ts_tablet_manager.cc:1918] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:17.362509 6771 log.cc:1196] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/8feb1a49db134429978387f18fe54865
I20260925 14:58:17.362960 6772 tablet_replica.cc:330] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: stopping tablet replica
I20260925 14:58:17.363003 6771 ts_tablet_manager.cc:1939] T 8feb1a49db134429978387f18fe54865 P e91f97e8f533437a8fe1ededb7891df1: Deleting consensus metadata
I20260925 14:58:17.363145 6772 raft_consensus.cc:2229] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:17.363385 6772 raft_consensus.cc:2258] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:17.364306 6772 ts_tablet_manager.cc:1905] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:17.364537 6185 catalog_manager.cc:4841] TS e91f97e8f533437a8fe1ededb7891df1 (127.4.243.193:45209): tablet 8feb1a49db134429978387f18fe54865 (table test-table [id=7d311f4eb49e4eeab73f6b7097bfed89]) successfully deleted
I20260925 14:58:17.370826 6772 ts_tablet_manager.cc:1918] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:17.370945 6772 log.cc:1196] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/8feb1a49db134429978387f18fe54865
I20260925 14:58:17.370939 6186 catalog_manager.cc:4841] TS 76c6c0a00bee40a585c7475a678f4b6d (127.4.243.195:41123): tablet 8feb1a49db134429978387f18fe54865 (table test-table [id=7d311f4eb49e4eeab73f6b7097bfed89]) successfully deleted
I20260925 14:58:17.371356 6772 ts_tablet_manager.cc:1939] T 8feb1a49db134429978387f18fe54865 P 308a1a32bc9b4d289dc9cfc17bc77554: Deleting consensus metadata
I20260925 14:58:17.372272 6183 catalog_manager.cc:4841] TS 308a1a32bc9b4d289dc9cfc17bc77554 (127.4.243.194:35873): tablet 8feb1a49db134429978387f18fe54865 (table test-table [id=7d311f4eb49e4eeab73f6b7097bfed89]) successfully deleted
Sep 25 14:58:17 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:17.383795 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.363845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.4.243.193:45209)
0925 14:58:17.364008 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.364015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.367121 (+ 3106us) server_negotiation.cc:407] Connection header received
0925 14:58:17.367307 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.367317 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.367373 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.367479 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:17.368785 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.369855 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.371705 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.372106 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.372215 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.377335 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:17.377350 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:17.377368 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:17.377411 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:17.380990 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.381671 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.381680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.381684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.381837 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.382251 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.382258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.382261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.382713 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:17.382989 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.383240 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.383435 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 14:58:17.384479 6400 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46708
I20260925 14:58:17.402832 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.387910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.4.243.254:44487)
0925 14:58:17.388101 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.388108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.388278 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:58:17.388452 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.388455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.388515 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.388590 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:17.390074 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.390854 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.392531 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.393165 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.393299 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.396010 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:17.396016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:17.396021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:17.396056 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:17.399626 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.400837 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.400861 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.400865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.401032 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.401398 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.401403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.401407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.401765 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:17.401995 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.402298 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.402510 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 14:58:17.407126 6199 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37202
I20260925 14:58:17.408777 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6270
I20260925 14:58:17.428074 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.409958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.4.243.193:45209)
0925 14:58:17.410148 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.410154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.415493 (+ 5339us) server_negotiation.cc:407] Connection header received
0925 14:58:17.415645 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.415651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.415726 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.415829 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:17.417204 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.418229 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.420171 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.420829 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.420967 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.423270 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:17.423275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:17.423278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:17.423307 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:17.425604 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.426213 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.426224 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.426226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.426337 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.426774 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.426779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.426784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.427036 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:17.427277 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.427564 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.427775 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 14:58:17.428622 6400 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46714
W20260925 14:58:17.429184 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:17.429366 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 6270
I20260925 14:58:17.459412 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6409
Sep 25 14:58:17 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:17.476927 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.460191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.4.243.194:35873)
0925 14:58:17.460367 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.460373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.460391 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:58:17.460527 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.460532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.460610 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.460715 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:17.461799 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.462474 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.463779 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.464478 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.464608 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.469653 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:17.469667 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:17.469687 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:17.469730 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:17.473607 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.474436 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.474446 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.474450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.474714 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.475217 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.475223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.475227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.475661 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:17.475951 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.476316 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.476559 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 14:58:17.477640 6539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37668
W20260925 14:58:17.478232 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6409: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:17.478395 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 6409
I20260925 14:58:17.495242 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6548
Sep 25 14:58:17 dist-test-slave-qxh0 krb5kdc[6154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:17.511168 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.496046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.4.243.195:41123)
0925 14:58:17.496231 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.496236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.496256 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:58:17.496343 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.496348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.496438 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.496543 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:17.497627 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.498692 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.499855 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.500506 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.500624 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.504764 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:17.504781 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:17.504806 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:17.504847 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:17.508539 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.509113 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.509121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.509124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.509282 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.509606 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.509611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.509614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.509994 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:17.510312 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.510533 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.510755 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 14:58:17.511937 6678 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55356
W20260925 14:58:17.512570 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6548: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:17.512722 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 6548
I20260925 14:58:17.530970 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6167
I20260925 14:58:17.532099 6239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:17.643882 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 6167
2026-09-25T14:58:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4894 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:17 dist-test-slave-qxh0 krb5kdc[6782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:17 dist-test-slave-qxh0 krb5kdc[6782](info): set up 2 sockets
Sep 25 14:58:17 dist-test-slave-qxh0 krb5kdc[6782](info): commencing operation
krb5kdc: starting...
W20260925 14:58:18.722681 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:18 dist-test-slave-qxh0 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:18.893261 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:45893
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:45893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:19.194917 6795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:19.195166 6795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:19.195343 6795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:19.202064 6795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:19.202163 6795 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:58:19.202229 6795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:19.202345 6795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:19.202420 6795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:19.202517 6795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:19.211555 6795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:45893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:19.212658 6795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:19.213955 6795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:19.225219 6800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:19.225261 6801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:19.225229 6803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:19.226297 6795 server_base.cc:1034] running on GCE node
I20260925 14:58:19.227041 6795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:19.228283 6795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:19.229521 6795 hybrid_clock.cc:648] HybridClock initialized: now 1790348299229471 us; error 49 us; skew 500 ppm
I20260925 14:58:19.229823 6795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:19 dist-test-slave-qxh0 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348299, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:19.232849 6795 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:19.234193 6795 webserver.cc:460] Webserver started at http://127.4.243.254:42673/ using document root <none> and password file <none>
I20260925 14:58:19.234921 6795 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:19.235042 6795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:19.235375 6795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:19.238118 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "d0cc2e2d80e5438dba5d10278df2b7b5"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-qxh0"
server_key: "bea95d90691faf3e97bc43edb858270c"
server_key_iv: "2a05cfe1a2ed817c7908e9a7b46923a1"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.238912 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "d0cc2e2d80e5438dba5d10278df2b7b5"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-qxh0"
server_key: "bea95d90691faf3e97bc43edb858270c"
server_key_iv: "2a05cfe1a2ed817c7908e9a7b46923a1"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.243643 6795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 14:58:19.246876 6810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:19.248142 6795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:58:19.248268 6795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "d0cc2e2d80e5438dba5d10278df2b7b5"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-qxh0"
server_key: "bea95d90691faf3e97bc43edb858270c"
server_key_iv: "2a05cfe1a2ed817c7908e9a7b46923a1"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.248431 6795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:19.269676 6795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:19.273594 6795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:19.273813 6795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:19.294819 6795 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:45893
I20260925 14:58:19.294838 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:45893 every 8 connection(s)
I20260925 14:58:19.296198 6795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:19.297610 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 6795
I20260925 14:58:19.297811 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:19.298581 5071 external_mini_cluster.cc:1428] Setting key 948377ba43358514bd9669c792720d26
I20260925 14:58:19.301069 6872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:58:19 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:19.313789 6872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Bootstrap starting.
I20260925 14:58:19.317785 6872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:19.319113 6872 log.cc:824] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:19.321853 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:19.300612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.4.243.254:45893)
0925 14:58:19.301408 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:19.301419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:19.301458 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:58:19.303086 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:19.303106 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:19.303466 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:19.303780 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:19.305459 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.306848 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.308541 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.309155 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.309278 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:19.313747 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:19.313771 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:19.313790 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:19.313837 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:19.317521 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.318206 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.318214 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.318222 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.318449 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.318835 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.318843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.318846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.319485 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:19.319821 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:19.320196 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 14:58:19.320577 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":387,"threads_started":1}
I20260925 14:58:19.322333 6872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: No bootstrap required, opened a new log
I20260925 14:58:19.329846 6872 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45893 } }
I20260925 14:58:19.330137 6872 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:19.330209 6872 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0cc2e2d80e5438dba5d10278df2b7b5, State: Initialized, Role: FOLLOWER
I20260925 14:58:19.330703 6872 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45893 } }
I20260925 14:58:19.330886 6872 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:19.330977 6872 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:19.331090 6872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:19.333022 6872 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45893 } }
I20260925 14:58:19.333448 6872 leader_election.cc:302] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0cc2e2d80e5438dba5d10278df2b7b5; no voters:
I20260925 14:58:19.333952 6872 leader_election.cc:288] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:19.334084 6877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:19.334815 6877 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [term 1 LEADER]: Becoming Leader. State: Replica: d0cc2e2d80e5438dba5d10278df2b7b5, State: Running, Role: LEADER
I20260925 14:58:19.335390 6877 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45893 } }
I20260925 14:58:19.336112 6872 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:19.338359 6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45893 } } }
I20260925 14:58:19.338829 6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:19.339285 6879 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0cc2e2d80e5438dba5d10278df2b7b5. Latest consensus state: current_term: 1 leader_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0cc2e2d80e5438dba5d10278df2b7b5" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45893 } } }
I20260925 14:58:19.339650 6879 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5 [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:19.348498 6893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:19.348598 6893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:19.348747 6886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:19.350852 6886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:19.359272 6886 catalog_manager.cc:1347] Generated new cluster ID: cf04388e04b44b05bcd5382acc17eca1
I20260925 14:58:19.359366 6886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:19.380369 6886 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:19.381789 6886 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:19.390805 6886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Generated new TSK 0
I20260925 14:58:19.391582 6886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:19.440546 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45893
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:19.755812 6898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:19.756067 6898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:19.756223 6898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:19.763235 6898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:19.763362 6898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:19.763507 6898 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:58:19.763566 6898 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:19.763684 6898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:19.773007 6898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:19.774223 6898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:19.775592 6898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:19.788232 6903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:19.788282 6904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:19.788304 6906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:19.789099 6898 server_base.cc:1034] running on GCE node
I20260925 14:58:19.789700 6898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:19.790529 6898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:19.791795 6898 hybrid_clock.cc:648] HybridClock initialized: now 1790348299791720 us; error 47 us; skew 500 ppm
I20260925 14:58:19.792117 6898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:19 dist-test-slave-qxh0 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348299, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:19.795140 6898 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:19.796523 6898 webserver.cc:460] Webserver started at http://127.4.243.193:41115/ using document root <none> and password file <none>
I20260925 14:58:19.797173 6898 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:19.797317 6898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:19.797617 6898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:19.800402 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "857c08e5461840e68934dd94108c4a40"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-qxh0"
server_key: "27f0d15d1cbe8241e21144bc496644da"
server_key_iv: "e1337c71de7c7b59fe7e77c684679fba"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.801141 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "857c08e5461840e68934dd94108c4a40"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-qxh0"
server_key: "27f0d15d1cbe8241e21144bc496644da"
server_key_iv: "e1337c71de7c7b59fe7e77c684679fba"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.806011 6898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 14:58:19.809281 6913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:19.810516 6898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:19.810717 6898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "857c08e5461840e68934dd94108c4a40"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-qxh0"
server_key: "27f0d15d1cbe8241e21144bc496644da"
server_key_iv: "e1337c71de7c7b59fe7e77c684679fba"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.810905 6898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:19.825889 6898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:19.829743 6898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:19.830024 6898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:19.832617 6898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:19.832755 6898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:19.832909 6898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:19.833015 6898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:19 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348299, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:19.858369 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:19.833653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45893 (local address 127.4.243.193:40533)
0925 14:58:19.834497 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:19.834539 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:19.836718 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:19.837441 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:19.837454 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:19.838236 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:19.839191 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:19.839225 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.840971 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.840980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:19.843145 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:19.843157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.844114 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.844121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:19.844323 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:19.845112 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:19.845133 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:19.848382 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:19.852405 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:19.852419 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:19.852449 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:19.853116 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:19.853508 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:19.853513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:19.853517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:19.853848 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:19.854535 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:19.854548 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:19.855146 (+ 598us) client_negotiation.cc:769] Sending connection context
0925 14:58:19.855539 (+ 393us) client_negotiation.cc:240] Negotiation successful
0925 14:58:19.856287 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":345,"threads_started":1}
I20260925 14:58:19.859189 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:19.832974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:40533 (local address 127.4.243.254:45893)
0925 14:58:19.833458 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:19.833464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:19.834675 (+ 1211us) server_negotiation.cc:407] Connection header received
0925 14:58:19.836985 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:19.836992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:19.837085 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:19.837197 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:19.839402 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.840764 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.843305 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.843886 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.844027 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:19.848683 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:19.848691 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:19.848704 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:19.848747 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:19.852191 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.853254 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.853257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.853260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.853371 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.854026 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.854031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.854034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.854381 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:19.854536 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:19.858426 (+ 3890us) server_negotiation.cc:299] Negotiation successful
0925 14:58:19.858673 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":199,"threads_started":1}
I20260925 14:58:19.882313 6898 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:39033
I20260925 14:58:19.882383 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:39033 every 8 connection(s)
I20260925 14:58:19.884013 6898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:19.886395 7034 heartbeater.cc:347] Connected to a master server at 127.4.243.254:45893
I20260925 14:58:19.886626 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 6898
I20260925 14:58:19.886656 7034 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:19.886824 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:19.887425 7034 heartbeater.cc:511] Master 127.4.243.254:45893 requested a full tablet report, sending...
I20260925 14:58:19.887609 5071 external_mini_cluster.cc:1428] Setting key 0ddafb773694a86bc83b6e96634c6ef0
I20260925 14:58:19.889655 6827 ts_manager.cc:194] Registered new tserver with Master: 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033)
I20260925 14:58:19.890234 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:19.873391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45893 (local address 127.4.243.193:37213)
0925 14:58:19.874114 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:19.874136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:19.874351 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:19.874754 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:19.874760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:19.875512 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:19.876160 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:19.876178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.877880 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.877888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:19.880238 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:19.880258 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.881539 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.881549 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:19.881770 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:19.882652 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:19.882673 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:19.883662 (+ 989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:19.887281 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:19.887288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:19.887296 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:19.887825 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:19.888403 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:19.888411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:19.888416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:19.888594 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:19.889201 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:19.889206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:19.889395 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 14:58:19.889550 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 14:58:19.889841 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":404,"threads_started":1}
I20260925 14:58:19.891101 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:19.873919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:37213 (local address 127.4.243.254:45893)
0925 14:58:19.874104 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:19.874111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:19.874237 (+ 126us) server_negotiation.cc:407] Connection header received
0925 14:58:19.874494 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:19.874497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:19.874555 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:19.874720 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:19.876375 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.877636 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.880442 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.881096 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.881214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:19.883834 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:19.883838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:19.883841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:19.883868 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:19.887141 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.887952 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.887957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.887961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.888105 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.888702 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.888707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.888710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.889064 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:19.889200 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:19.890504 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0925 14:58:19.890760 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:58:19.891584 6827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:40533
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:19.927884 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45893
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:20.240388 7038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:20.240648 7038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:20.240741 7038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:20.247814 7038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:20.247918 7038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:20.248000 7038 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:58:20.248051 7038 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:20.248109 7038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:20.257465 7038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:20.258689 7038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:20.259996 7038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:20.272289 7043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:20.272313 7044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:20.272563 7046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:20.273027 7038 server_base.cc:1034] running on GCE node
I20260925 14:58:20.273622 7038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:20.274417 7038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:20.275604 7038 hybrid_clock.cc:648] HybridClock initialized: now 1790348300275547 us; error 51 us; skew 500 ppm
I20260925 14:58:20.275921 7038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:20 dist-test-slave-qxh0 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:20.278928 7038 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:20.280366 7038 webserver.cc:460] Webserver started at http://127.4.243.194:43135/ using document root <none> and password file <none>
I20260925 14:58:20.281015 7038 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:20.281157 7038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:20.281458 7038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:20.284444 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "b51c383f610a4272835851370a244c8e"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-qxh0"
server_key: "1177d00cf40ef10a139ed7a0aaf5a882"
server_key_iv: "f96288a53374351ebd41f53330235fcf"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.285211 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "b51c383f610a4272835851370a244c8e"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-qxh0"
server_key: "1177d00cf40ef10a139ed7a0aaf5a882"
server_key_iv: "f96288a53374351ebd41f53330235fcf"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.290246 7038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:58:20.293707 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:20.295176 7038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:20.295328 7038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "b51c383f610a4272835851370a244c8e"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-qxh0"
server_key: "1177d00cf40ef10a139ed7a0aaf5a882"
server_key_iv: "f96288a53374351ebd41f53330235fcf"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.295506 7038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:20.317139 7038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:20.321295 7038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:20.321584 7038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:20.324416 7038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:20.324551 7038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:20.324687 7038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:20.324786 7038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:20 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:20.347867 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.325042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:35401 (local address 127.4.243.254:45893)
0925 14:58:20.325251 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.325258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.326724 (+ 1466us) server_negotiation.cc:407] Connection header received
0925 14:58:20.328798 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.328803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.328909 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.329022 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:20.331116 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.332237 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.334907 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.335552 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.335685 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.340037 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:20.340044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:20.340049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:20.340092 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:20.343508 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.344702 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.344708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.344712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.344866 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.345474 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.345480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.345486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.345804 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:20.345962 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.347018 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.347343 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 14:58:20.348430 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.325663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45893 (local address 127.4.243.194:35401)
0925 14:58:20.326534 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:20.326588 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:20.328465 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:20.329253 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:20.329261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:20.329971 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:20.330918 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.330945 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.332408 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.332415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.334765 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.334778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.335814 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.335820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.336134 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.336954 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:20.336977 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:20.339810 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:20.343721 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.343733 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.343755 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.344535 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.344998 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.345010 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.345017 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.345325 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.345942 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:20.345951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:20.346484 (+ 533us) client_negotiation.cc:769] Sending connection context
0925 14:58:20.346880 (+ 396us) client_negotiation.cc:240] Negotiation successful
0925 14:58:20.347297 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":376,"threads_started":1}
I20260925 14:58:20.371697 7172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:38289 every 8 connection(s)
I20260925 14:58:20.371697 7038 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:38289
I20260925 14:58:20.373394 7038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:20.374135 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 7038
I20260925 14:58:20.374308 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:20.374881 5071 external_mini_cluster.cc:1428] Setting key 3b5dfa26de24db2039b4fd8a80df82a8
I20260925 14:58:20.375662 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.355347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45893 (local address 127.4.243.194:56961)
0925 14:58:20.356058 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:20.356081 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:20.356330 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:20.356782 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:20.356790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:20.357524 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:20.358038 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.358057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.359438 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.359444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.361162 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.361180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.362383 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.362391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.362627 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.367878 (+ 5251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:20.367909 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:20.368963 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:20.372879 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.372882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.372885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.373239 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.373620 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.373623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.373625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.373770 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.374539 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:20.374547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:20.374775 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 14:58:20.374903 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 14:58:20.375216 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":343,"threads_started":1}
I20260925 14:58:20.376645 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.355384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:56961 (local address 127.4.243.254:45893)
0925 14:58:20.355616 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.355624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.356172 (+ 548us) server_negotiation.cc:407] Connection header received
0925 14:58:20.356580 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.356584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.356638 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.356722 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:20.358277 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.359298 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.361328 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.362135 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.362347 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.369263 (+ 6916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:20.369272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:20.369279 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:20.369323 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:20.372757 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.373360 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.373369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.373373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.373523 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.373916 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.373921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.373924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.374392 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:20.376020 (+ 1628us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.376135 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.376353 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:58:20.377526 7173 heartbeater.cc:347] Connected to a master server at 127.4.243.254:45893
I20260925 14:58:20.377880 7173 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:20.379153 7173 heartbeater.cc:511] Master 127.4.243.254:45893 requested a full tablet report, sending...
I20260925 14:58:20.380828 6827 ts_manager.cc:194] Registered new tserver with Master: b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:20.381762 6827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:35401
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:20.418439 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:45893
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:20.728737 7177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:20.729046 7177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:20.729211 7177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:20.736161 7177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:20.736305 7177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:20.736455 7177 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:58:20.736552 7177 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:20.736671 7177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:20.746049 7177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:20.747265 7177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:20.748557 7177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:20.761125 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:20.761160 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:20.761196 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:20.762113 7177 server_base.cc:1034] running on GCE node
I20260925 14:58:20.762720 7177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:20.763622 7177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:20.764832 7177 hybrid_clock.cc:648] HybridClock initialized: now 1790348300764780 us; error 49 us; skew 500 ppm
I20260925 14:58:20.765108 7177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:20 dist-test-slave-qxh0 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:20.768005 7177 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:20.769337 7177 webserver.cc:460] Webserver started at http://127.4.243.195:44625/ using document root <none> and password file <none>
I20260925 14:58:20.769968 7177 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:20.770104 7177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:20.770407 7177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:20.773149 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "cb02701c70d3498388dfaed28264e809"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-qxh0"
server_key: "e1bced9563423d19f99adcf44e9524ce"
server_key_iv: "6b756b5528ce3cd56f45e914726655b5"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.773871 7177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "cb02701c70d3498388dfaed28264e809"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-qxh0"
server_key: "e1bced9563423d19f99adcf44e9524ce"
server_key_iv: "6b756b5528ce3cd56f45e914726655b5"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.778923 7177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 14:58:20.782162 7192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:20.783442 7177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:20.783607 7177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "cb02701c70d3498388dfaed28264e809"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-qxh0"
server_key: "e1bced9563423d19f99adcf44e9524ce"
server_key_iv: "6b756b5528ce3cd56f45e914726655b5"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.783790 7177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:20.808061 7177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:20.811877 7177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:20.812150 7177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:20.814841 7177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:20.814989 7177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:20.815135 7177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:20.815277 7177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:20 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:20.839147 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.815546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:51011 (local address 127.4.243.254:45893)
0925 14:58:20.815749 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.815755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.817257 (+ 1502us) server_negotiation.cc:407] Connection header received
0925 14:58:20.819559 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.819566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.819655 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.819739 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:20.822199 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.823383 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.825635 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.826253 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.826430 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.831540 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:20.831545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:20.831549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:20.831580 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:20.835335 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.836305 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.836314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.836319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.836520 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.836989 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.836997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.837003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.837472 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:20.837649 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.838378 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.838689 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:58:20.840123 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.816177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45893 (local address 127.4.243.195:51011)
0925 14:58:20.817035 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:20.817087 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:20.819163 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:20.820304 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:20.820311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:20.821025 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:20.822016 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.822046 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.823561 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.823565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.825481 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.825492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.826552 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.826567 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.826960 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.827914 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:20.827951 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:20.831319 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:20.835497 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.835510 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.835530 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.836174 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.836654 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.836657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.836659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.836879 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.837651 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:20.837657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:20.837983 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 14:58:20.838217 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 14:58:20.838724 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":381,"threads_started":1}
I20260925 14:58:20.864517 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.849524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:41559 (local address 127.4.243.254:45893)
0925 14:58:20.849750 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.849758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.850427 (+ 669us) server_negotiation.cc:407] Connection header received
0925 14:58:20.850627 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.850634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.850743 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.850859 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:20.851938 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.853401 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.854806 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.855551 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.855683 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.857363 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:20.857372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:20.857379 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:20.857425 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:20.861215 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.862277 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.862285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.862291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.862436 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.862931 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.862936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.862939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.863298 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:20.863455 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.863841 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.864099 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:58:20.864684 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.849594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45893 (local address 127.4.243.195:41559)
0925 14:58:20.850304 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:20.850334 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:20.850485 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:20.850954 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:20.850961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:20.851450 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:20.851805 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.851815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.853531 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.853535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.854661 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.854683 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.855767 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.855772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.855887 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.856499 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:20.856516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:20.857164 (+ 648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:20.861417 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.861423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.861429 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.862110 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.862614 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:20.862622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:20.862628 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:20.862809 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:20.863420 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:20.863423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:20.863576 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 14:58:20.863701 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 14:58:20.864045 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":381,"threads_started":1}
I20260925 14:58:20.871240 7177 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42111
I20260925 14:58:20.871316 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42111 every 8 connection(s)
I20260925 14:58:20.872434 7177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:20.874480 7312 heartbeater.cc:347] Connected to a master server at 127.4.243.254:45893
I20260925 14:58:20.874728 7312 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:20.875375 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 7177
I20260925 14:58:20.875499 7312 heartbeater.cc:511] Master 127.4.243.254:45893 requested a full tablet report, sending...
I20260925 14:58:20.875557 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:20.876114 5071 external_mini_cluster.cc:1428] Setting key cb96c7bf49681733d3b0f6de64bf0ee4
I20260925 14:58:20.876880 6827 ts_manager.cc:194] Registered new tserver with Master: cb02701c70d3498388dfaed28264e809 (127.4.243.195:42111)
I20260925 14:58:20.877805 6827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:51011
I20260925 14:58:20.879084 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:20.894851 7034 heartbeater.cc:503] Master 127.4.243.254:45893 was elected leader, sending a full tablet report...
Sep 25 14:58:20 dist-test-slave-qxh0 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:20 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:20.924122 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.910722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.4.243.254:45893)
0925 14:58:20.910940 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.910947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.911163 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:58:20.911298 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.911302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.911372 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.911453 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:20.912717 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.913534 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.915162 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.915647 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.915766 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.919415 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:20.919420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:20.919424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:20.919454 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:20.921662 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.922338 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.922342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.922344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.922454 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.922878 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.922882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.922884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.923156 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:20.923343 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.923600 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.923765 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 14:58:20.928174 6826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:20.935024 6826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:20.957532 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.947067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42111 (local address 127.0.0.1:34240)
0925 14:58:20.948072 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:20.948093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:20.948214 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:20.949533 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:20.949538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:20.949560 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:20.950028 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.950039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.952279 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.952284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.954754 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.954771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.956529 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.956535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.956734 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.956784 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:58:20.956908 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:58:20.957067 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"spinlock_wait_cycles":19840,"thread_start_us":901,"threads_started":1}
I20260925 14:58:20.958276 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.946814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.4.243.193:39033)
0925 14:58:20.947467 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.947478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.948051 (+ 573us) server_negotiation.cc:407] Connection header received
0925 14:58:20.948458 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.948469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.948728 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.948897 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:20.950041 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.951847 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.955485 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.956877 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.957016 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.957219 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.957851 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.958005 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":370,"threads_started":1}
I20260925 14:58:20.958616 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.947765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.4.243.195:42111)
0925 14:58:20.948847 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.948854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.948883 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:58:20.948975 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.948983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.949305 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.950218 (+ 913us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:20.950512 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.952099 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.954934 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.956347 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.956483 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.956722 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.958112 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.958280 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":730,"threads_started":1}
I20260925 14:58:20.959098 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.946723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39033 (local address 127.0.0.1:58856)
0925 14:58:20.947895 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:20.947954 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:20.948214 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:20.949151 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:20.949271 (+ 120us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:20.949310 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:20.949801 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.949812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.952006 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.952011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.955316 (+ 3305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.955333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.957370 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.957376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.957621 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.957671 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:58:20.958522 (+ 851us) client_negotiation.cc:240] Negotiation successful
0925 14:58:20.958705 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":938,"thread_start_us":760,"threads_started":1}
I20260925 14:58:20.962066 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.948826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:38289 (local address 127.0.0.1:58274)
0925 14:58:20.954045 (+ 5219us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:20.954067 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:20.954184 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:20.955490 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:20.955494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:20.955519 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:20.955861 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.955870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.957420 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.957425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.959314 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:20.959330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.961255 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.961261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:20.961426 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.961457 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 14:58:20.961575 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 14:58:20.961700 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5088,"mutex_wait_us":22,"thread_start_us":4954,"threads_started":1}
I20260925 14:58:20.962255 7242 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2787e56f4f45bbb985e0c4fcf4d090 (DEFAULT_TABLE table=test-table [id=ad9613fc113b4d4fa83d67c669bd189a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:20.962849 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.948568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.4.243.194:38289)
0925 14:58:20.953653 (+ 5085us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.953661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.954308 (+ 647us) server_negotiation.cc:407] Connection header received
0925 14:58:20.954395 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.954401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.954670 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.954880 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:20.956021 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.957259 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.959483 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.960739 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.960941 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.961133 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.962377 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.962536 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4926,"thread_start_us":4801,"threads_started":1}
I20260925 14:58:20.962721 6964 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2787e56f4f45bbb985e0c4fcf4d090 (DEFAULT_TABLE table=test-table [id=ad9613fc113b4d4fa83d67c669bd189a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:20.963761 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2787e56f4f45bbb985e0c4fcf4d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:20.964061 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2787e56f4f45bbb985e0c4fcf4d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:20.966298 7103 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2787e56f4f45bbb985e0c4fcf4d090 (DEFAULT_TABLE table=test-table [id=ad9613fc113b4d4fa83d67c669bd189a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:20.967592 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2787e56f4f45bbb985e0c4fcf4d090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:20.975004 7331 tablet_bootstrap.cc:492] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Bootstrap starting.
I20260925 14:58:20.977844 7332 tablet_bootstrap.cc:492] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Bootstrap starting.
I20260925 14:58:20.978575 7331 tablet_bootstrap.cc:654] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:20.979926 7331 log.cc:824] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:20.980907 7333 tablet_bootstrap.cc:492] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Bootstrap starting.
I20260925 14:58:20.981336 7332 tablet_bootstrap.cc:654] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:20.982402 7332 log.cc:824] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:20.983888 7331 tablet_bootstrap.cc:492] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: No bootstrap required, opened a new log
I20260925 14:58:20.984130 7331 ts_tablet_manager.cc:1397] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 14:58:20.984285 7333 tablet_bootstrap.cc:654] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:20.985179 7333 log.cc:824] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:20.985605 7332 tablet_bootstrap.cc:492] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: No bootstrap required, opened a new log
I20260925 14:58:20.985846 7332 ts_tablet_manager.cc:1397] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 14:58:20.992430 7333 tablet_bootstrap.cc:492] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: No bootstrap required, opened a new log
I20260925 14:58:20.992671 7333 ts_tablet_manager.cc:1397] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260925 14:58:20.994655 7331 raft_consensus.cc:348] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:20.995054 7331 raft_consensus.cc:374] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:20.995144 7331 raft_consensus.cc:729] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb02701c70d3498388dfaed28264e809, State: Initialized, Role: FOLLOWER
I20260925 14:58:20.995739 7331 consensus_queue.cc:260] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:20.996146 7332 raft_consensus.cc:348] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:20.996562 7332 raft_consensus.cc:374] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:20.996654 7332 raft_consensus.cc:729] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857c08e5461840e68934dd94108c4a40, State: Initialized, Role: FOLLOWER
I20260925 14:58:20.996873 7312 heartbeater.cc:503] Master 127.4.243.254:45893 was elected leader, sending a full tablet report...
I20260925 14:58:20.997210 7332 consensus_queue.cc:260] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:20.998875 7332 ts_tablet_manager.cc:1428] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 14:58:20.999136 7331 ts_tablet_manager.cc:1428] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260925 14:58:21.003181 7333 raft_consensus.cc:348] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.003453 7333 raft_consensus.cc:374] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:21.003559 7333 raft_consensus.cc:729] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51c383f610a4272835851370a244c8e, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.004001 7333 consensus_queue.cc:260] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.004879 7173 heartbeater.cc:503] Master 127.4.243.254:45893 was elected leader, sending a full tablet report...
I20260925 14:58:21.005360 7333 ts_tablet_manager.cc:1428] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260925 14:58:21.026993 7337 raft_consensus.cc:482] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:21.027236 7337 raft_consensus.cc:504] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.028767 7337 leader_election.cc:288] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:21.035612 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.029035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:41543 (local address 127.4.243.193:39033)
0925 14:58:21.029213 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.029221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.029310 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:58:21.029566 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.029574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.029673 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.029790 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:21.030212 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.031245 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.033605 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.034864 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.035032 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.035106 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.035299 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.035423 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:58:21.035698 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.028981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39033 (local address 127.4.243.195:41543)
0925 14:58:21.029212 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.029233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.029388 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.029811 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.029821 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.029847 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:21.030062 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.030074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.031393 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.031397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.033419 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.033433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.034993 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.034997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.035146 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.035169 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.035245 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.035420 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:58:21.036223 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.029460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:49585 (local address 127.4.243.194:38289)
0925 14:58:21.029642 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.029650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.030084 (+ 434us) server_negotiation.cc:407] Connection header received
0925 14:58:21.030229 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.030234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.030325 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.030414 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:21.030900 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.031743 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.033995 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.035197 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.035371 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.035433 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.035857 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.035986 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:58:21.036249 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.029360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:38289 (local address 127.4.243.195:49585)
0925 14:58:21.029982 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.030000 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.030115 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.030427 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.030430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.030448 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:21.030771 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.030779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.031879 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.031884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.033831 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.033843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.035413 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.035418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.035578 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.035608 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.035729 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.035860 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":294,"threads_started":1}
I20260925 14:58:21.036710 6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2787e56f4f45bbb985e0c4fcf4d090" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40" is_pre_election: true
I20260925 14:58:21.037276 6984 raft_consensus.cc:2454] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:21.037351 7128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2787e56f4f45bbb985e0c4fcf4d090" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e" is_pre_election: true
I20260925 14:58:21.037796 7128 raft_consensus.cc:2454] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:21.037844 7194 leader_election.cc:302] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:21.038210 7337 raft_consensus.cc:2781] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:21.038341 7337 raft_consensus.cc:482] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:21.038465 7337 raft_consensus.cc:3037] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.040786 7337 raft_consensus.cc:504] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.041427 7337 leader_election.cc:288] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Requested vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:21.041779 6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2787e56f4f45bbb985e0c4fcf4d090" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40"
I20260925 14:58:21.041865 7128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2787e56f4f45bbb985e0c4fcf4d090" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e"
I20260925 14:58:21.041952 6984 raft_consensus.cc:3037] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.042080 7128 raft_consensus.cc:3037] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.044278 6984 raft_consensus.cc:2454] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:21.044302 7128 raft_consensus.cc:2454] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:21.044611 7194 leader_election.cc:302] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:21.044862 7337 raft_consensus.cc:2781] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:21.045212 7337 raft_consensus.cc:686] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Becoming Leader. State: Replica: cb02701c70d3498388dfaed28264e809, State: Running, Role: LEADER
I20260925 14:58:21.045666 7337 consensus_queue.cc:237] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.049850 6826 catalog_manager.cc:5495] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb02701c70d3498388dfaed28264e809 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } health_report { overall_health: HEALTHY } } }
Sep 25 14:58:21 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348299, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:21.093746 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.080887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.4.243.254:45893)
0925 14:58:21.081048 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.081053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.081581 (+ 528us) server_negotiation.cc:407] Connection header received
0925 14:58:21.081772 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.081778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.081846 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.081948 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:21.083052 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.083733 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.085022 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.085446 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.085569 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.088661 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:21.088674 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:21.088678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:21.088704 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:21.091216 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.091869 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.091875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.091881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.092003 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.092349 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.092356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.092360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.092599 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:21.092744 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.093114 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.093400 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":54}
I20260925 14:58:21.093777 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.080949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:45893 (local address 127.0.0.1:54948)
0925 14:58:21.081484 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.081500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.081660 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.081970 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.081974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.082664 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:21.082916 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.082925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.083852 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.083859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.084899 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.084910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.085625 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.085641 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.085791 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.086373 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:21.086392 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:21.088496 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:21.091350 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.091361 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.091369 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.091758 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.092113 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.092117 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.092121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.092251 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.092723 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:21.092731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:21.092895 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.093039 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.093284 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":278,"threads_started":1}
I20260925 14:58:21.098876 6826 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:54948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:58:21.099936 6826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:21.106137 7103 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73424ec3cb4699afe63afdb197a1e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58f0289b972e4365a17a080f6e2927eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:21.106027 6964 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73424ec3cb4699afe63afdb197a1e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58f0289b972e4365a17a080f6e2927eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:21.107025 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73424ec3cb4699afe63afdb197a1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:21.107025 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73424ec3cb4699afe63afdb197a1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:21.114069 7332 tablet_bootstrap.cc:492] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: Bootstrap starting.
I20260925 14:58:21.114135 7333 tablet_bootstrap.cc:492] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: Bootstrap starting.
I20260925 14:58:21.115612 7332 tablet_bootstrap.cc:654] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:21.115060 7242 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73424ec3cb4699afe63afdb197a1e6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58f0289b972e4365a17a080f6e2927eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:21.115764 7333 tablet_bootstrap.cc:654] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:21.116040 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73424ec3cb4699afe63afdb197a1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:21.117695 7332 tablet_bootstrap.cc:492] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: No bootstrap required, opened a new log
I20260925 14:58:21.117851 7332 ts_tablet_manager.cc:1397] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:58:21.118431 7332 raft_consensus.cc:348] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.118659 7332 raft_consensus.cc:374] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:21.118773 7332 raft_consensus.cc:729] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857c08e5461840e68934dd94108c4a40, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.119001 7332 consensus_queue.cc:260] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.119567 7332 ts_tablet_manager.cc:1428] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:21.119410 7338 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.119798 7338 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:21.121047 7333 tablet_bootstrap.cc:492] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: No bootstrap required, opened a new log
I20260925 14:58:21.121179 7333 ts_tablet_manager.cc:1397] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 14:58:21.121850 7333 raft_consensus.cc:348] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.122077 7333 raft_consensus.cc:374] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:21.122162 7333 raft_consensus.cc:729] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51c383f610a4272835851370a244c8e, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.122387 7333 consensus_queue.cc:260] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.122876 7339 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.123240 7339 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:21.123661 7333 ts_tablet_manager.cc:1428] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:21.123854 7331 tablet_bootstrap.cc:492] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: Bootstrap starting.
W20260925 14:58:21.123944 7313 tablet.cc:2369] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:21.125485 7331 tablet_bootstrap.cc:654] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:21.127563 7331 tablet_bootstrap.cc:492] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: No bootstrap required, opened a new log
I20260925 14:58:21.127688 7331 ts_tablet_manager.cc:1397] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:58:21.128293 7331 raft_consensus.cc:348] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.128502 7331 raft_consensus.cc:374] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:21.128603 7331 raft_consensus.cc:729] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb02701c70d3498388dfaed28264e809, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.128816 7331 consensus_queue.cc:260] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.129207 7337 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.129365 7331 ts_tablet_manager.cc:1428] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:58:21.129463 7337 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 14:58:21.131353 7174 tablet.cc:2369] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:21.137179 7035 tablet.cc:2369] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:21.230376 7337 raft_consensus.cc:482] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:21.230638 7337 raft_consensus.cc:504] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.231271 7337 leader_election.cc:288] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:21.231848 6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73424ec3cb4699afe63afdb197a1e6" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40" is_pre_election: true
I20260925 14:58:21.231910 7128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73424ec3cb4699afe63afdb197a1e6" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e" is_pre_election: true
I20260925 14:58:21.232118 6984 raft_consensus.cc:2454] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:21.232188 7128 raft_consensus.cc:2454] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:21.232545 7194 leader_election.cc:302] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:21.232827 7337 raft_consensus.cc:2781] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:21.232923 7337 raft_consensus.cc:482] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:21.233035 7337 raft_consensus.cc:3037] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.235059 7337 raft_consensus.cc:504] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.235639 7337 leader_election.cc:288] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Requested vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:21.236004 6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73424ec3cb4699afe63afdb197a1e6" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40"
I20260925 14:58:21.236135 7128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73424ec3cb4699afe63afdb197a1e6" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e"
I20260925 14:58:21.236205 6984 raft_consensus.cc:3037] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.236346 7128 raft_consensus.cc:3037] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.239104 6984 raft_consensus.cc:2454] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:21.239173 7128 raft_consensus.cc:2454] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:21.239459 7194 leader_election.cc:302] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:21.239709 7337 raft_consensus.cc:2781] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:21.239893 7337 raft_consensus.cc:686] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Becoming Leader. State: Replica: cb02701c70d3498388dfaed28264e809, State: Running, Role: LEADER
I20260925 14:58:21.240139 7337 consensus_queue.cc:237] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.241788 7341 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: TxnStatusTablet state changed. Reason: New leader cb02701c70d3498388dfaed28264e809. Latest consensus state: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.242132 7341 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:21.242935 7353 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:58:21.243273 7353 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:58:21.243570 6826 catalog_manager.cc:5495] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb02701c70d3498388dfaed28264e809 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:21.276402 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.270548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.4.243.195:42111)
0925 14:58:21.270789 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.270795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.270883 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:58:21.271149 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.271152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.271226 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.271338 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:21.271731 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.272428 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.274248 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.274723 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.274818 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.275402 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:21.275876 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:21.275975 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.276126 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.276188 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 14:58:21.276566 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.270544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42111 (local address 127.0.0.1:34242)
0925 14:58:21.270792 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.270812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.270952 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.271384 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.271389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.271402 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:21.271588 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.271598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.272548 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.272554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.274090 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.274104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.274871 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.274878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.275086 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.275186 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:21.275981 (+ 795us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:21.276007 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.276097 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.276247 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 14:58:21.283726 6984 raft_consensus.cc:1261] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:21.283998 7128 raft_consensus.cc:1261] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:21.284607 7337 consensus_queue.cc:1035] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:21.285030 7341 consensus_queue.cc:1035] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:21.292402 7339 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: TxnStatusTablet state changed. Reason: New leader cb02701c70d3498388dfaed28264e809. Latest consensus state: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.292680 7339 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:21.310948 7338 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: TxnStatusTablet state changed. Reason: New leader cb02701c70d3498388dfaed28264e809. Latest consensus state: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.311237 7338 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:21.318156 7337 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.318446 7337 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:21.318976 7338 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.319185 7338 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P 857c08e5461840e68934dd94108c4a40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:21.319259 7337 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.319500 7337 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P cb02701c70d3498388dfaed28264e809: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:21.322929 7339 tablet_replica.cc:439] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } } }
I20260925 14:58:21.323160 7339 tablet_replica.cc:442] T 3a73424ec3cb4699afe63afdb197a1e6 P b51c383f610a4272835851370a244c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:21.333616 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.327285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.4.243.195:42111)
0925 14:58:21.327539 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.327543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.327787 (+ 244us) server_negotiation.cc:407] Connection header received
0925 14:58:21.327851 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.327857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.327932 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.328014 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:21.328545 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.329764 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.331728 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.332370 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.332516 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.332870 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:21.333010 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:21.333094 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.333262 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.333343 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260925 14:58:21.349249 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.344307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:45929 (local address 127.4.243.195:42111)
0925 14:58:21.344497 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.344502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.344688 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:58:21.344871 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.344875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.344941 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.345034 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:21.345407 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.346099 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.347627 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.348029 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.348165 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.348542 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:21.348652 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:21.348749 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.348878 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.348940 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":44}
I20260925 14:58:21.349323 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.344404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42111 (local address 127.4.243.195:45929)
0925 14:58:21.344603 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.344617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.344754 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.345057 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.345065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.345075 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:21.345283 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.345291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.346256 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.346261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.347493 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.347504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.348170 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.348175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.348312 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.348356 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:21.348748 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:21.348759 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.348863 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.349003 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":33}
I20260925 14:58:21.358333 7128 raft_consensus.cc:1261] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:21.358466 6983 raft_consensus.cc:1261] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:21.358973 7337 consensus_queue.cc:1035] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:21.359217 7341 consensus_queue.cc:1035] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:21.363593 7356 mvcc.cc:205] Tried to move back new op lower bound from 7333266642359427072 to 7333266641086525440. Current Snapshot: MvccSnapshot[applied={T|T < 7333266642359427072}]
I20260925 14:58:21.364734 7358 mvcc.cc:205] Tried to move back new op lower bound from 7333266642359427072 to 7333266641086525440. Current Snapshot: MvccSnapshot[applied={T|T < 7333266642359427072}]
I20260925 14:58:21.381369 7357 mvcc.cc:205] Tried to move back new op lower bound from 7333266642359427072 to 7333266641086525440. Current Snapshot: MvccSnapshot[applied={T|T < 7333266642359427072}]
I20260925 14:58:21.438344 6826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:21.438696 6826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:21.442312 6826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Sending DeleteTablet for 3 replicas of tablet 5a2787e56f4f45bbb985e0c4fcf4d090
I20260925 14:58:21.443555 6964 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2787e56f4f45bbb985e0c4fcf4d090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:21 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58856
I20260925 14:58:21.443539 7103 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2787e56f4f45bbb985e0c4fcf4d090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:21 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58274
I20260925 14:58:21.444985 6826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:21.445957 6826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:21.447690 7242 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2787e56f4f45bbb985e0c4fcf4d090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:21 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:34240
I20260925 14:58:21.448474 7372 tablet_replica.cc:330] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: stopping tablet replica
I20260925 14:58:21.448979 7372 raft_consensus.cc:2229] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:21.449427 7372 raft_consensus.cc:2258] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:21.452231 7372 ts_tablet_manager.cc:1905] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:21.452955 7373 tablet_replica.cc:330] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: stopping tablet replica
I20260925 14:58:21.453146 7374 tablet_replica.cc:330] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: stopping tablet replica
I20260925 14:58:21.453533 7374 raft_consensus.cc:2229] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:21.454130 7374 raft_consensus.cc:2258] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:21.454761 7103 tablet_service.cc:1467] Processing CreateTablet for tablet 601ef5d1ce2f4127a6de1d49344f0f00 (DEFAULT_TABLE table=test-table [id=666f2b6b65434a5ba225c8cd09c76f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:21.454856 6964 tablet_service.cc:1467] Processing CreateTablet for tablet 601ef5d1ce2f4127a6de1d49344f0f00 (DEFAULT_TABLE table=test-table [id=666f2b6b65434a5ba225c8cd09c76f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:21.455291 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601ef5d1ce2f4127a6de1d49344f0f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:21.455430 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601ef5d1ce2f4127a6de1d49344f0f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:21.455765 7242 tablet_service.cc:1467] Processing CreateTablet for tablet 601ef5d1ce2f4127a6de1d49344f0f00 (DEFAULT_TABLE table=test-table [id=666f2b6b65434a5ba225c8cd09c76f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:21.456188 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601ef5d1ce2f4127a6de1d49344f0f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:21.462358 7373 raft_consensus.cc:2229] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:21.462817 7373 raft_consensus.cc:2258] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:21.463438 7332 tablet_bootstrap.cc:492] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: Bootstrap starting.
I20260925 14:58:21.464979 7332 tablet_bootstrap.cc:654] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:21.465656 7331 tablet_bootstrap.cc:492] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: Bootstrap starting.
I20260925 14:58:21.466245 7333 tablet_bootstrap.cc:492] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: Bootstrap starting.
I20260925 14:58:21.467448 7331 tablet_bootstrap.cc:654] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:21.467849 7333 tablet_bootstrap.cc:654] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:21.470541 7374 ts_tablet_manager.cc:1905] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:21.476353 7373 ts_tablet_manager.cc:1905] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:21.480114 7331 tablet_bootstrap.cc:492] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: No bootstrap required, opened a new log
I20260925 14:58:21.480319 7331 ts_tablet_manager.cc:1397] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260925 14:58:21.480664 7372 ts_tablet_manager.cc:1918] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:21.480844 7372 log.cc:1196] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/5a2787e56f4f45bbb985e0c4fcf4d090
I20260925 14:58:21.481324 7372 ts_tablet_manager.cc:1939] T 5a2787e56f4f45bbb985e0c4fcf4d090 P b51c383f610a4272835851370a244c8e: Deleting consensus metadata
I20260925 14:58:21.481278 7331 raft_consensus.cc:348] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.481583 7331 raft_consensus.cc:374] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:21.481667 7331 raft_consensus.cc:729] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb02701c70d3498388dfaed28264e809, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.481998 7331 consensus_queue.cc:260] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.482606 7331 ts_tablet_manager.cc:1428] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:58:21.485395 7333 tablet_bootstrap.cc:492] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: No bootstrap required, opened a new log
I20260925 14:58:21.485559 7333 ts_tablet_manager.cc:1397] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20260925 14:58:21.486303 7374 ts_tablet_manager.cc:1918] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:21.486436 7374 log.cc:1196] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/5a2787e56f4f45bbb985e0c4fcf4d090
I20260925 14:58:21.486963 7374 ts_tablet_manager.cc:1939] T 5a2787e56f4f45bbb985e0c4fcf4d090 P cb02701c70d3498388dfaed28264e809: Deleting consensus metadata
I20260925 14:58:21.486397 7333 raft_consensus.cc:348] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.487164 7373 ts_tablet_manager.cc:1918] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:21.487236 7333 raft_consensus.cc:374] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:21.487324 7373 log.cc:1196] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/5a2787e56f4f45bbb985e0c4fcf4d090
I20260925 14:58:21.487326 7333 raft_consensus.cc:729] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51c383f610a4272835851370a244c8e, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.487689 7333 consensus_queue.cc:260] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.488399 7332 tablet_bootstrap.cc:492] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: No bootstrap required, opened a new log
I20260925 14:58:21.488556 7332 ts_tablet_manager.cc:1397] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20260925 14:58:21.489233 7332 raft_consensus.cc:348] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.489464 7332 raft_consensus.cc:374] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:21.489552 7332 raft_consensus.cc:729] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857c08e5461840e68934dd94108c4a40, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.489784 7332 consensus_queue.cc:260] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.490365 7332 ts_tablet_manager.cc:1428] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:21.490525 6811 catalog_manager.cc:4841] TS cb02701c70d3498388dfaed28264e809 (127.4.243.195:42111): tablet 5a2787e56f4f45bbb985e0c4fcf4d090 (table test-table [id=ad9613fc113b4d4fa83d67c669bd189a]) successfully deleted
I20260925 14:58:21.491530 7373 ts_tablet_manager.cc:1939] T 5a2787e56f4f45bbb985e0c4fcf4d090 P 857c08e5461840e68934dd94108c4a40: Deleting consensus metadata
I20260925 14:58:21.492008 7333 ts_tablet_manager.cc:1428] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260925 14:58:21.492689 6812 catalog_manager.cc:4841] TS b51c383f610a4272835851370a244c8e (127.4.243.194:38289): tablet 5a2787e56f4f45bbb985e0c4fcf4d090 (table test-table [id=ad9613fc113b4d4fa83d67c669bd189a]) successfully deleted
I20260925 14:58:21.492949 6812 catalog_manager.cc:4841] TS 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033): tablet 5a2787e56f4f45bbb985e0c4fcf4d090 (table test-table [id=ad9613fc113b4d4fa83d67c669bd189a]) successfully deleted
I20260925 14:58:21.599629 7369 raft_consensus.cc:482] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:21.599887 7369 raft_consensus.cc:504] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.600678 7369 leader_election.cc:288] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:21.601231 6983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601ef5d1ce2f4127a6de1d49344f0f00" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40" is_pre_election: true
I20260925 14:58:21.601291 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601ef5d1ce2f4127a6de1d49344f0f00" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e" is_pre_election: true
I20260925 14:58:21.601488 6983 raft_consensus.cc:2454] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:21.601549 7127 raft_consensus.cc:2454] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:21.601971 7194 leader_election.cc:302] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b51c383f610a4272835851370a244c8e, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:21.602255 7369 raft_consensus.cc:2781] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:21.602375 7369 raft_consensus.cc:482] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:21.602499 7369 raft_consensus.cc:3037] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.604674 7369 raft_consensus.cc:504] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.605280 7369 leader_election.cc:288] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Requested vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:21.605659 6983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601ef5d1ce2f4127a6de1d49344f0f00" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40"
I20260925 14:58:21.605787 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601ef5d1ce2f4127a6de1d49344f0f00" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e"
I20260925 14:58:21.605870 6983 raft_consensus.cc:3037] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.605995 7127 raft_consensus.cc:3037] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:21.608481 6983 raft_consensus.cc:2454] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:21.608881 7194 leader_election.cc:302] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:21.609071 7127 raft_consensus.cc:2454] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:21.609200 7369 raft_consensus.cc:2781] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:21.609365 7369 raft_consensus.cc:686] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Becoming Leader. State: Replica: cb02701c70d3498388dfaed28264e809, State: Running, Role: LEADER
I20260925 14:58:21.609649 7369 consensus_queue.cc:237] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:21.612314 6827 catalog_manager.cc:5495] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb02701c70d3498388dfaed28264e809 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:21.664892 7128 raft_consensus.cc:1261] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:21.664888 6983 raft_consensus.cc:1261] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:21.665547 7369 consensus_queue.cc:1035] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:21.665944 7341 consensus_queue.cc:1035] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:58:21.688568 7356 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:58:23.263622 6827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:23.263832 6827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:23.267445 6827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Sending DeleteTablet for 3 replicas of tablet 601ef5d1ce2f4127a6de1d49344f0f00
I20260925 14:58:23.268211 6964 tablet_service.cc:1514] Processing DeleteTablet for tablet 601ef5d1ce2f4127a6de1d49344f0f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:23 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58856
I20260925 14:58:23.268263 7242 tablet_service.cc:1514] Processing DeleteTablet for tablet 601ef5d1ce2f4127a6de1d49344f0f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:23 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:34240
I20260925 14:58:23.268332 7103 tablet_service.cc:1514] Processing DeleteTablet for tablet 601ef5d1ce2f4127a6de1d49344f0f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:23 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58274
I20260925 14:58:23.280517 7400 tablet_replica.cc:330] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: stopping tablet replica
I20260925 14:58:23.280757 7402 tablet_replica.cc:330] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: stopping tablet replica
I20260925 14:58:23.280756 7400 raft_consensus.cc:2229] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:23.280999 7400 raft_consensus.cc:2258] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:23.281016 7401 tablet_replica.cc:330] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: stopping tablet replica
I20260925 14:58:23.281128 7402 raft_consensus.cc:2229] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:23.281402 7402 raft_consensus.cc:2258] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:23.281739 7401 raft_consensus.cc:2229] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:23.281926 7400 ts_tablet_manager.cc:1905] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:23.282101 7401 raft_consensus.cc:2258] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:23.282630 7402 ts_tablet_manager.cc:1905] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:23.282951 7401 ts_tablet_manager.cc:1905] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:23.288880 7400 ts_tablet_manager.cc:1918] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:23.289132 7400 log.cc:1196] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/601ef5d1ce2f4127a6de1d49344f0f00
I20260925 14:58:23.289362 7401 ts_tablet_manager.cc:1918] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:23.289489 7401 log.cc:1196] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/601ef5d1ce2f4127a6de1d49344f0f00
I20260925 14:58:23.289582 7400 ts_tablet_manager.cc:1939] T 601ef5d1ce2f4127a6de1d49344f0f00 P 857c08e5461840e68934dd94108c4a40: Deleting consensus metadata
I20260925 14:58:23.289939 7401 ts_tablet_manager.cc:1939] T 601ef5d1ce2f4127a6de1d49344f0f00 P cb02701c70d3498388dfaed28264e809: Deleting consensus metadata
I20260925 14:58:23.290642 7402 ts_tablet_manager.cc:1918] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:23.290728 6812 catalog_manager.cc:4841] TS 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033): tablet 601ef5d1ce2f4127a6de1d49344f0f00 (table test-table [id=666f2b6b65434a5ba225c8cd09c76f25]) successfully deleted
I20260925 14:58:23.290824 7402 log.cc:1196] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/601ef5d1ce2f4127a6de1d49344f0f00
I20260925 14:58:23.290902 6811 catalog_manager.cc:4841] TS cb02701c70d3498388dfaed28264e809 (127.4.243.195:42111): tablet 601ef5d1ce2f4127a6de1d49344f0f00 (table test-table [id=666f2b6b65434a5ba225c8cd09c76f25]) successfully deleted
I20260925 14:58:23.291373 7402 ts_tablet_manager.cc:1939] T 601ef5d1ce2f4127a6de1d49344f0f00 P b51c383f610a4272835851370a244c8e: Deleting consensus metadata
I20260925 14:58:23.292268 6812 catalog_manager.cc:4841] TS b51c383f610a4272835851370a244c8e (127.4.243.194:38289): tablet 601ef5d1ce2f4127a6de1d49344f0f00 (table test-table [id=666f2b6b65434a5ba225c8cd09c76f25]) successfully deleted
Sep 25 14:58:23 dist-test-slave-qxh0 krb5kdc[6782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:23 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:23.332996 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:23.320048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.4.243.254:45893)
0925 14:58:23.320591 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:23.320596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:23.320615 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:58:23.320733 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:23.320736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:23.320794 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:23.320880 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:23.322151 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.322912 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.324197 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.324599 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.324696 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:23.328386 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:23.328391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:23.328394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:23.328423 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:23.330945 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.331481 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.331484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.331487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.331609 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.331959 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.331962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.331964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.332186 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:23.332339 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:23.332569 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 14:58:23.332714 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
I20260925 14:58:23.336439 6827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:23.337091 6827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:23.343307 6964 tablet_service.cc:1467] Processing CreateTablet for tablet 81b95bb808a54423a310020f233cee83 (DEFAULT_TABLE table=test-table [id=6f016b97d7cf4b85b8c5397926b636c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:23.343461 7103 tablet_service.cc:1467] Processing CreateTablet for tablet 81b95bb808a54423a310020f233cee83 (DEFAULT_TABLE table=test-table [id=6f016b97d7cf4b85b8c5397926b636c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:23.343729 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b95bb808a54423a310020f233cee83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:23.343640 7242 tablet_service.cc:1467] Processing CreateTablet for tablet 81b95bb808a54423a310020f233cee83 (DEFAULT_TABLE table=test-table [id=6f016b97d7cf4b85b8c5397926b636c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:23.343884 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b95bb808a54423a310020f233cee83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:23.344000 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b95bb808a54423a310020f233cee83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:23.350052 7411 tablet_bootstrap.cc:492] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: Bootstrap starting.
I20260925 14:58:23.350736 7412 tablet_bootstrap.cc:492] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: Bootstrap starting.
I20260925 14:58:23.352381 7411 tablet_bootstrap.cc:654] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:23.352946 7412 tablet_bootstrap.cc:654] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:23.355412 7412 tablet_bootstrap.cc:492] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: No bootstrap required, opened a new log
I20260925 14:58:23.355592 7412 ts_tablet_manager.cc:1397] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 14:58:23.356577 7412 raft_consensus.cc:348] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.356889 7412 raft_consensus.cc:374] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:23.356984 7412 raft_consensus.cc:729] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb02701c70d3498388dfaed28264e809, State: Initialized, Role: FOLLOWER
I20260925 14:58:23.357304 7412 consensus_queue.cc:260] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.357913 7412 ts_tablet_manager.cc:1428] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:23.358479 7413 tablet_bootstrap.cc:492] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: Bootstrap starting.
I20260925 14:58:23.360072 7413 tablet_bootstrap.cc:654] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:23.361788 7411 tablet_bootstrap.cc:492] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: No bootstrap required, opened a new log
I20260925 14:58:23.361920 7411 ts_tablet_manager.cc:1397] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260925 14:58:23.362673 7411 raft_consensus.cc:348] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.362929 7411 raft_consensus.cc:374] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:23.363046 7411 raft_consensus.cc:729] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51c383f610a4272835851370a244c8e, State: Initialized, Role: FOLLOWER
I20260925 14:58:23.363255 7411 consensus_queue.cc:260] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.364013 7411 ts_tablet_manager.cc:1428] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:23.365242 7413 tablet_bootstrap.cc:492] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: No bootstrap required, opened a new log
I20260925 14:58:23.365398 7413 ts_tablet_manager.cc:1397] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 14:58:23.366465 7413 raft_consensus.cc:348] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.366720 7413 raft_consensus.cc:374] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:23.366822 7413 raft_consensus.cc:729] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857c08e5461840e68934dd94108c4a40, State: Initialized, Role: FOLLOWER
I20260925 14:58:23.367062 7413 consensus_queue.cc:260] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.367838 7413 ts_tablet_manager.cc:1428] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:23.489151 7383 raft_consensus.cc:482] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:23.489375 7383 raft_consensus.cc:504] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.490067 7383 leader_election.cc:288] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:23.490710 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b95bb808a54423a310020f233cee83" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e" is_pre_election: true
I20260925 14:58:23.490721 6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b95bb808a54423a310020f233cee83" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40" is_pre_election: true
I20260925 14:58:23.490959 7127 raft_consensus.cc:2454] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:23.491003 6984 raft_consensus.cc:2454] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:23.491463 7194 leader_election.cc:302] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b51c383f610a4272835851370a244c8e, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:23.491734 7383 raft_consensus.cc:2781] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:23.491871 7383 raft_consensus.cc:482] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:23.491997 7383 raft_consensus.cc:3037] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:23.493988 7383 raft_consensus.cc:504] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.494467 7383 leader_election.cc:288] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Requested vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:23.495005 6984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b95bb808a54423a310020f233cee83" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40"
I20260925 14:58:23.495031 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b95bb808a54423a310020f233cee83" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e"
I20260925 14:58:23.495187 6984 raft_consensus.cc:3037] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:23.495695 7127 raft_consensus.cc:3037] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:23.497668 6984 raft_consensus.cc:2454] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:23.498172 7127 raft_consensus.cc:2454] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:23.498257 7194 leader_election.cc:302] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:23.498618 7383 raft_consensus.cc:2781] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:23.498778 7383 raft_consensus.cc:686] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Becoming Leader. State: Replica: cb02701c70d3498388dfaed28264e809, State: Running, Role: LEADER
I20260925 14:58:23.499105 7383 consensus_queue.cc:237] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.501497 6826 catalog_manager.cc:5495] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb02701c70d3498388dfaed28264e809 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:23.551563 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:23.546300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.4.243.195:42111)
0925 14:58:23.546852 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:23.546856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:23.546874 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:58:23.546917 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:23.546921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:23.547000 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:23.547078 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:23.547415 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.548203 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.549449 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.550245 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.550336 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:23.550838 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:23.550933 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:23.551003 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:23.551123 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 14:58:23.551174 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":355,"threads_started":1}
I20260925 14:58:23.559540 7127 raft_consensus.cc:1261] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:23.559609 6984 raft_consensus.cc:1261] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:23.560075 7394 consensus_queue.cc:1035] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:23.560407 7383 consensus_queue.cc:1035] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:23.601200 6827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:23.601428 6827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:23.604166 6827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Sending DeleteTablet for 3 replicas of tablet 81b95bb808a54423a310020f233cee83
I20260925 14:58:23.604861 6964 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b95bb808a54423a310020f233cee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:23 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58856
I20260925 14:58:23.604888 7103 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b95bb808a54423a310020f233cee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:23 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58274
I20260925 14:58:23.605129 7400 tablet_replica.cc:330] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: stopping tablet replica
I20260925 14:58:23.605090 7242 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b95bb808a54423a310020f233cee83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:23 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:34240
I20260925 14:58:23.605223 7402 tablet_replica.cc:330] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: stopping tablet replica
I20260925 14:58:23.605326 7400 raft_consensus.cc:2229] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:23.605458 7402 raft_consensus.cc:2229] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:23.605492 7400 raft_consensus.cc:2258] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:23.605640 7402 raft_consensus.cc:2258] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:23.605890 7401 tablet_replica.cc:330] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: stopping tablet replica
I20260925 14:58:23.606114 7401 raft_consensus.cc:2229] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:23.606379 7401 raft_consensus.cc:2258] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:23.606437 7400 ts_tablet_manager.cc:1905] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:23.606650 6827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:23.607214 6827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:23.607417 7401 ts_tablet_manager.cc:1905] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:23.613270 7400 ts_tablet_manager.cc:1918] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:23.613404 7400 log.cc:1196] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/81b95bb808a54423a310020f233cee83
I20260925 14:58:23.613344 7103 tablet_service.cc:1467] Processing CreateTablet for tablet 6971b35f14a6480eb3adcea4c7a285a1 (DEFAULT_TABLE table=test-table [id=045493a5848d4bdda64f6944f05f08ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:23.613394 7242 tablet_service.cc:1467] Processing CreateTablet for tablet 6971b35f14a6480eb3adcea4c7a285a1 (DEFAULT_TABLE table=test-table [id=045493a5848d4bdda64f6944f05f08ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:23.613767 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6971b35f14a6480eb3adcea4c7a285a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:23.613796 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6971b35f14a6480eb3adcea4c7a285a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:23.613809 7400 ts_tablet_manager.cc:1939] T 81b95bb808a54423a310020f233cee83 P 857c08e5461840e68934dd94108c4a40: Deleting consensus metadata
I20260925 14:58:23.617795 7412 tablet_bootstrap.cc:492] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: Bootstrap starting.
I20260925 14:58:23.618286 6812 catalog_manager.cc:4841] TS 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033): tablet 81b95bb808a54423a310020f233cee83 (table test-table [id=6f016b97d7cf4b85b8c5397926b636c9]) successfully deleted
I20260925 14:58:23.618772 7411 tablet_bootstrap.cc:492] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: Bootstrap starting.
I20260925 14:58:23.618710 6964 tablet_service.cc:1467] Processing CreateTablet for tablet 6971b35f14a6480eb3adcea4c7a285a1 (DEFAULT_TABLE table=test-table [id=045493a5848d4bdda64f6944f05f08ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:23.619108 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6971b35f14a6480eb3adcea4c7a285a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:23.619498 7412 tablet_bootstrap.cc:654] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:23.620302 7401 ts_tablet_manager.cc:1918] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:23.620324 7411 tablet_bootstrap.cc:654] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:23.620432 7401 log.cc:1196] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/81b95bb808a54423a310020f233cee83
I20260925 14:58:23.621908 7401 ts_tablet_manager.cc:1939] T 81b95bb808a54423a310020f233cee83 P cb02701c70d3498388dfaed28264e809: Deleting consensus metadata
I20260925 14:58:23.623909 7402 ts_tablet_manager.cc:1905] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:23.626427 7413 tablet_bootstrap.cc:492] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: Bootstrap starting.
I20260925 14:58:23.628086 7413 tablet_bootstrap.cc:654] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:23.628090 7412 tablet_bootstrap.cc:492] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: No bootstrap required, opened a new log
I20260925 14:58:23.628333 7412 ts_tablet_manager.cc:1397] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20260925 14:58:23.628446 6811 catalog_manager.cc:4841] TS cb02701c70d3498388dfaed28264e809 (127.4.243.195:42111): tablet 81b95bb808a54423a310020f233cee83 (table test-table [id=6f016b97d7cf4b85b8c5397926b636c9]) successfully deleted
I20260925 14:58:23.629196 7412 raft_consensus.cc:348] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.629515 7412 raft_consensus.cc:374] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:23.629652 7412 raft_consensus.cc:729] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb02701c70d3498388dfaed28264e809, State: Initialized, Role: FOLLOWER
I20260925 14:58:23.629951 7412 consensus_queue.cc:260] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.631222 7402 ts_tablet_manager.cc:1918] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:23.631371 7402 log.cc:1196] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/81b95bb808a54423a310020f233cee83
I20260925 14:58:23.631783 7402 ts_tablet_manager.cc:1939] T 81b95bb808a54423a310020f233cee83 P b51c383f610a4272835851370a244c8e: Deleting consensus metadata
I20260925 14:58:23.632735 6812 catalog_manager.cc:4841] TS b51c383f610a4272835851370a244c8e (127.4.243.194:38289): tablet 81b95bb808a54423a310020f233cee83 (table test-table [id=6f016b97d7cf4b85b8c5397926b636c9]) successfully deleted
I20260925 14:58:23.633026 7412 ts_tablet_manager.cc:1428] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260925 14:58:23.635612 7411 tablet_bootstrap.cc:492] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: No bootstrap required, opened a new log
I20260925 14:58:23.635779 7411 ts_tablet_manager.cc:1397] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260925 14:58:23.636540 7411 raft_consensus.cc:348] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.636751 7411 raft_consensus.cc:374] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:23.636842 7411 raft_consensus.cc:729] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51c383f610a4272835851370a244c8e, State: Initialized, Role: FOLLOWER
I20260925 14:58:23.637087 7411 consensus_queue.cc:260] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.637570 7411 ts_tablet_manager.cc:1428] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:58:23.639950 7413 tablet_bootstrap.cc:492] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: No bootstrap required, opened a new log
I20260925 14:58:23.640120 7413 ts_tablet_manager.cc:1397] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260925 14:58:23.640910 7413 raft_consensus.cc:348] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.641113 7413 raft_consensus.cc:374] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:23.641662 7413 raft_consensus.cc:729] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857c08e5461840e68934dd94108c4a40, State: Initialized, Role: FOLLOWER
I20260925 14:58:23.641922 7413 consensus_queue.cc:260] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.642356 7413 ts_tablet_manager.cc:1428] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:58:23.802033 7383 raft_consensus.cc:482] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:23.802205 7383 raft_consensus.cc:504] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.802914 7383 leader_election.cc:288] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:23.803462 6983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6971b35f14a6480eb3adcea4c7a285a1" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40" is_pre_election: true
I20260925 14:58:23.803512 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6971b35f14a6480eb3adcea4c7a285a1" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e" is_pre_election: true
I20260925 14:58:23.803730 6983 raft_consensus.cc:2454] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:23.803768 7127 raft_consensus.cc:2454] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 0.
I20260925 14:58:23.804194 7194 leader_election.cc:302] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:23.804490 7383 raft_consensus.cc:2781] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:23.804585 7383 raft_consensus.cc:482] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:23.804713 7383 raft_consensus.cc:3037] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:23.806725 7383 raft_consensus.cc:504] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.807224 7383 leader_election.cc:288] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Requested vote from peers 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033), b51c383f610a4272835851370a244c8e (127.4.243.194:38289)
I20260925 14:58:23.807814 6983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6971b35f14a6480eb3adcea4c7a285a1" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857c08e5461840e68934dd94108c4a40"
I20260925 14:58:23.807816 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6971b35f14a6480eb3adcea4c7a285a1" candidate_uuid: "cb02701c70d3498388dfaed28264e809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51c383f610a4272835851370a244c8e"
I20260925 14:58:23.808037 6983 raft_consensus.cc:3037] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:23.808033 7127 raft_consensus.cc:3037] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:23.810958 7127 raft_consensus.cc:2454] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:23.810957 6983 raft_consensus.cc:2454] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb02701c70d3498388dfaed28264e809 in term 1.
I20260925 14:58:23.811358 7194 leader_election.cc:302] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857c08e5461840e68934dd94108c4a40, cb02701c70d3498388dfaed28264e809; no voters:
I20260925 14:58:23.811625 7383 raft_consensus.cc:2781] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:23.811793 7383 raft_consensus.cc:686] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Becoming Leader. State: Replica: cb02701c70d3498388dfaed28264e809, State: Running, Role: LEADER
I20260925 14:58:23.812076 7383 consensus_queue.cc:237] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } }
I20260925 14:58:23.814769 6827 catalog_manager.cc:5495] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb02701c70d3498388dfaed28264e809 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "cb02701c70d3498388dfaed28264e809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb02701c70d3498388dfaed28264e809" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42111 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:23.878300 7127 raft_consensus.cc:1261] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:23.878338 6983 raft_consensus.cc:1261] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Refusing update from remote peer cb02701c70d3498388dfaed28264e809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:23.878890 7394 consensus_queue.cc:1035] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51c383f610a4272835851370a244c8e" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 38289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:23.879228 7341 consensus_queue.cc:1035] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857c08e5461840e68934dd94108c4a40" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:58:23.898563 7381 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:58:25.486470 6827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:25.486727 6827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:25.490428 6827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0cc2e2d80e5438dba5d10278df2b7b5: Sending DeleteTablet for 3 replicas of tablet 6971b35f14a6480eb3adcea4c7a285a1
I20260925 14:58:25.491158 6964 tablet_service.cc:1514] Processing DeleteTablet for tablet 6971b35f14a6480eb3adcea4c7a285a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58856
I20260925 14:58:25.491251 7242 tablet_service.cc:1514] Processing DeleteTablet for tablet 6971b35f14a6480eb3adcea4c7a285a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:34240
I20260925 14:58:25.491689 7103 tablet_service.cc:1514] Processing DeleteTablet for tablet 6971b35f14a6480eb3adcea4c7a285a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58274
I20260925 14:58:25.492372 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6898
I20260925 14:58:25.504346 7452 tablet_replica.cc:330] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: stopping tablet replica
I20260925 14:58:25.504767 7452 raft_consensus.cc:2229] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:25.505177 7452 raft_consensus.cc:2258] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.506515 7452 ts_tablet_manager.cc:1905] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:25.513574 7454 tablet_replica.cc:330] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: stopping tablet replica
I20260925 14:58:25.513782 7454 raft_consensus.cc:2229] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:25.514048 7454 raft_consensus.cc:2258] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.514571 7453 tablet_replica.cc:330] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: stopping tablet replica
I20260925 14:58:25.514858 7453 raft_consensus.cc:2229] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:25.515196 7453 raft_consensus.cc:2258] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.515870 7452 ts_tablet_manager.cc:1918] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:25.515921 7454 ts_tablet_manager.cc:1905] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:25.515996 7452 log.cc:1196] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/6971b35f14a6480eb3adcea4c7a285a1
I20260925 14:58:25.516101 7453 ts_tablet_manager.cc:1905] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:25.516474 7452 ts_tablet_manager.cc:1939] T 6971b35f14a6480eb3adcea4c7a285a1 P 857c08e5461840e68934dd94108c4a40: Deleting consensus metadata
I20260925 14:58:25.517588 6812 catalog_manager.cc:4841] TS 857c08e5461840e68934dd94108c4a40 (127.4.243.193:39033): tablet 6971b35f14a6480eb3adcea4c7a285a1 (table test-table [id=045493a5848d4bdda64f6944f05f08ef]) successfully deleted
I20260925 14:58:25.521265 7454 ts_tablet_manager.cc:1918] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:25.521401 7454 log.cc:1196] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/6971b35f14a6480eb3adcea4c7a285a1
I20260925 14:58:25.521799 7454 ts_tablet_manager.cc:1939] T 6971b35f14a6480eb3adcea4c7a285a1 P b51c383f610a4272835851370a244c8e: Deleting consensus metadata
I20260925 14:58:25.521781 7453 ts_tablet_manager.cc:1918] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:25.521919 7453 log.cc:1196] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/6971b35f14a6480eb3adcea4c7a285a1
I20260925 14:58:25.522374 7453 ts_tablet_manager.cc:1939] T 6971b35f14a6480eb3adcea4c7a285a1 P cb02701c70d3498388dfaed28264e809: Deleting consensus metadata
I20260925 14:58:25.522756 6812 catalog_manager.cc:4841] TS b51c383f610a4272835851370a244c8e (127.4.243.194:38289): tablet 6971b35f14a6480eb3adcea4c7a285a1 (table test-table [id=045493a5848d4bdda64f6944f05f08ef]) successfully deleted
I20260925 14:58:25.523406 6811 catalog_manager.cc:4841] TS cb02701c70d3498388dfaed28264e809 (127.4.243.195:42111): tablet 6971b35f14a6480eb3adcea4c7a285a1 (table test-table [id=045493a5848d4bdda64f6944f05f08ef]) successfully deleted
Sep 25 14:58:25 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:25.529480 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.494490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.4.243.193:39033)
0925 14:58:25.500171 (+ 5681us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.500180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.506153 (+ 5973us) server_negotiation.cc:407] Connection header received
0925 14:58:25.506656 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.506665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.506776 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.506889 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.509212 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.510784 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.514308 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.515072 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.515184 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.524543 (+ 9359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.524559 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.524577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:25.524609 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.527221 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:25.527803 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:25.527810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:25.527812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:25.527929 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:25.528235 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:25.528237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:25.528239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:25.528549 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.528754 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.529083 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.529238 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5500,"thread_start_us":5359,"threads_started":1}
W20260925 14:58:25.530135 7029 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35984
W20260925 14:58:25.530792 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6898: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:25.530997 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 6898
I20260925 14:58:25.548677 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7038
Sep 25 14:58:25 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:25.564018 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.549385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.4.243.194:38289)
0925 14:58:25.549820 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.549825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.549841 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:58:25.549892 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.549895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.549951 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.550050 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.551504 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.552290 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.553537 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.553923 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.554080 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.558161 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.558175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.558191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:25.558222 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.561124 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:25.561826 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:25.561834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:25.561843 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:25.561979 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:25.562329 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:25.562334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:25.562337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:25.562883 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.563181 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.563453 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.563656 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
W20260925 14:58:25.564657 7168 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53840
W20260925 14:58:25.565294 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7038: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:25.565426 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 7038
I20260925 14:58:25.582712 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7177
Sep 25 14:58:25 dist-test-slave-qxh0 krb5kdc[6782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:25.597332 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.583437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.4.243.195:42111)
0925 14:58:25.583875 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.583879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.583895 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:58:25.583946 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.583950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.584022 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.584109 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.585284 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.586071 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.587386 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.587880 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.587974 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.592171 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.592182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.592199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:25.592232 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.594943 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:25.595565 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:25.595572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:25.595574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:25.595681 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:25.596017 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:25.596021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:25.596023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:25.596363 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.596591 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.596820 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.596970 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
W20260925 14:58:25.598040 7307 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58976
W20260925 14:58:25.598667 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:25.598803 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 7177
I20260925 14:58:25.622880 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 6795
I20260925 14:58:25.624042 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:25.738296 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 6795
2026-09-25T14:58:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8095 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:25 dist-test-slave-qxh0 krb5kdc[7462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:25 dist-test-slave-qxh0 krb5kdc[7462](info): set up 2 sockets
Sep 25 14:58:25 dist-test-slave-qxh0 krb5kdc[7462](info): commencing operation
krb5kdc: starting...
W20260925 14:58:26.862892 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:26 dist-test-slave-qxh0 krb5kdc[7462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:27.047175 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:39595
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:39595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:27.353760 7475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:27.354055 7475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:27.354218 7475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:27.361451 7475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:27.361600 7475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:27.361706 7475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:27.361825 7475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:27.361898 7475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:27.371208 7475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:39595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:27.372388 7475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:27.373639 7475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:27.385653 7483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:27.385666 7481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:27.385670 7475 server_base.cc:1034] running on GCE node
W20260925 14:58:27.385923 7480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:27.386864 7475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:27.388160 7475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:27.389436 7475 hybrid_clock.cc:648] HybridClock initialized: now 1790348307389381 us; error 36 us; skew 500 ppm
I20260925 14:58:27.389882 7475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:27 dist-test-slave-qxh0 krb5kdc[7462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348307, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:27.393025 7475 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:27.394316 7475 webserver.cc:460] Webserver started at http://127.4.243.254:40141/ using document root <none> and password file <none>
I20260925 14:58:27.394982 7475 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:27.395073 7475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:27.395304 7475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:27.398206 7475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "3d0054c33ebc4c3ca97d420b7053aa81"
format_stamp: "Formatted at 2026-09-25 14:58:27 on dist-test-slave-qxh0"
server_key: "fcf9479098ef88620b53a6669acc1114"
server_key_iv: "d1b3f4ff804b1912b254ca4a1dfceb60"
server_key_version: "encryptionkey@0"
I20260925 14:58:27.398959 7475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "3d0054c33ebc4c3ca97d420b7053aa81"
format_stamp: "Formatted at 2026-09-25 14:58:27 on dist-test-slave-qxh0"
server_key: "fcf9479098ef88620b53a6669acc1114"
server_key_iv: "d1b3f4ff804b1912b254ca4a1dfceb60"
server_key_version: "encryptionkey@0"
I20260925 14:58:27.403909 7475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 14:58:27.407043 7490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:27.408267 7475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:58:27.408416 7475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "3d0054c33ebc4c3ca97d420b7053aa81"
format_stamp: "Formatted at 2026-09-25 14:58:27 on dist-test-slave-qxh0"
server_key: "fcf9479098ef88620b53a6669acc1114"
server_key_iv: "d1b3f4ff804b1912b254ca4a1dfceb60"
server_key_version: "encryptionkey@0"
I20260925 14:58:27.408609 7475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:27.425619 7475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:27.429337 7475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:27.429601 7475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:27.447178 7475 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:39595
I20260925 14:58:27.447207 7541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:39595 every 8 connection(s)
I20260925 14:58:27.448453 7475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:27.452149 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:27.452915 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 7475
I20260925 14:58:27.453135 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:27.453735 5071 external_mini_cluster.cc:1428] Setting key d6d36dbab2c5a24821798c4cb0e63b3e
I20260925 14:58:27.465513 7542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81: Bootstrap starting.
Sep 25 14:58:27 dist-test-slave-qxh0 krb5kdc[7462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:27.469059 7542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:27.470207 7542 log.cc:824] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:27.472854 7542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81: No bootstrap required, opened a new log
I20260925 14:58:27.475576 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:27.455559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.4.243.254:39595)
0925 14:58:27.456313 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:27.456324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:27.456365 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:58:27.457908 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:27.457931 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:27.458286 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:27.458664 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:27.460397 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:27.461703 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:27.463559 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:27.464214 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:27.464345 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:27.468834 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:27.468842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:27.468855 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:27.468895 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:27.471872 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:27.472500 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:27.472505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:27.472511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:27.472656 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:27.473059 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:27.473063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:27.473064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:27.473508 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:27.473802 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:27.474166 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 14:58:27.474555 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":336,"threads_started":1}
I20260925 14:58:27.480872 7542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39595 } }
I20260925 14:58:27.481298 7542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:27.481408 7542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0054c33ebc4c3ca97d420b7053aa81, State: Initialized, Role: FOLLOWER
I20260925 14:58:27.481959 7542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39595 } }
I20260925 14:58:27.482178 7542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:27.482278 7542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:27.482388 7542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:27.484326 7542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39595 } }
I20260925 14:58:27.484710 7542 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0054c33ebc4c3ca97d420b7053aa81; no voters:
I20260925 14:58:27.485319 7542 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:27.485433 7547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:27.486146 7547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0054c33ebc4c3ca97d420b7053aa81, State: Running, Role: LEADER
I20260925 14:58:27.486683 7547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39595 } }
I20260925 14:58:27.487466 7542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:27.489436 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0054c33ebc4c3ca97d420b7053aa81. Latest consensus state: current_term: 1 leader_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39595 } } }
I20260925 14:58:27.489827 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:27.493737 7548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0054c33ebc4c3ca97d420b7053aa81" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39595 } } }
I20260925 14:58:27.494035 7548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:27.494711 7553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:27.498971 7553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:27.506662 7553 catalog_manager.cc:1347] Generated new cluster ID: 39f9ce22abc54a1c9b00bee7a81a9811
I20260925 14:58:27.506773 7553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:27.528435 7553 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:27.529671 7553 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:27.538012 7553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0054c33ebc4c3ca97d420b7053aa81: Generated new TSK 0
I20260925 14:58:27.538818 7553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:27.582783 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39595
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:27.892122 7568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:27.892395 7568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:27.892488 7568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:27.899526 7568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:27.899621 7568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:27.899725 7568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:27.909286 7568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:27.910422 7568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:27.911772 7568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:27.924866 7573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:27.924889 7574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:27.925383 7576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:27.927583 7568 server_base.cc:1034] running on GCE node
I20260925 14:58:27.928315 7568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:27.929360 7568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:27.930576 7568 hybrid_clock.cc:648] HybridClock initialized: now 1790348307930533 us; error 55 us; skew 500 ppm
I20260925 14:58:27.930908 7568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:27 dist-test-slave-qxh0 krb5kdc[7462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348307, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:27.933686 7568 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:27.935144 7568 webserver.cc:460] Webserver started at http://127.4.243.193:43185/ using document root <none> and password file <none>
I20260925 14:58:27.935853 7568 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:27.935997 7568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:27.936287 7568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:27.939177 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "01bc173840a24e0699adf276edef1faf"
format_stamp: "Formatted at 2026-09-25 14:58:27 on dist-test-slave-qxh0"
server_key: "01db18c721ae35f19ee50672dfedf1ca"
server_key_iv: "3482e3fc47a1f5b86f8c660f64b5d18e"
server_key_version: "encryptionkey@0"
I20260925 14:58:27.939926 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "01bc173840a24e0699adf276edef1faf"
format_stamp: "Formatted at 2026-09-25 14:58:27 on dist-test-slave-qxh0"
server_key: "01db18c721ae35f19ee50672dfedf1ca"
server_key_iv: "3482e3fc47a1f5b86f8c660f64b5d18e"
server_key_version: "encryptionkey@0"
I20260925 14:58:27.944804 7568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 14:58:27.948069 7583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:27.949322 7568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:27.949503 7568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "01bc173840a24e0699adf276edef1faf"
format_stamp: "Formatted at 2026-09-25 14:58:27 on dist-test-slave-qxh0"
server_key: "01db18c721ae35f19ee50672dfedf1ca"
server_key_iv: "3482e3fc47a1f5b86f8c660f64b5d18e"
server_key_version: "encryptionkey@0"
I20260925 14:58:27.949687 7568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:27.966199 7568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:27.969941 7568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:27.970216 7568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:27.971161 7568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:27.972398 7568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:27.972496 7568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:27.972640 7568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:27.972707 7568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.005755 7568 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:38047
I20260925 14:58:28.005795 7695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:38047 every 8 connection(s)
I20260925 14:58:28.007133 7568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:28.008730 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 7568
I20260925 14:58:28.008952 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:28.009656 5071 external_mini_cluster.cc:1428] Setting key 2bf132ed0b841fdbb4cf2c58f5c7dbe0
Sep 25 14:58:28 dist-test-slave-qxh0 krb5kdc[7462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348307, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:28.028595 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.009676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:49917 (local address 127.4.243.254:39595)
0925 14:58:28.010118 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:28.010124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:28.011162 (+ 1038us) server_negotiation.cc:407] Connection header received
0925 14:58:28.013026 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:28.013035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:28.013122 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:28.013258 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:28.014687 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.015867 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.017372 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.018118 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.018255 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.021587 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:28.021596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:28.021603 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:28.021645 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:28.025466 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.026220 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.026227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.026231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.026380 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.026852 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.026857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.026861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.027235 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:28.027394 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:28.028082 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 14:58:28.028316 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260925 14:58:28.029592 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.010222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39595 (local address 127.4.243.193:49917)
0925 14:58:28.010997 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:28.011046 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:28.012738 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:28.013454 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:28.013462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:28.013952 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:28.014488 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.014505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.016017 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.016021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.017245 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.017256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.018321 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.018326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.018517 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.019264 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:28.019285 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:28.021379 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:28.025638 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.025645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.025658 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.026089 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.026499 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.026502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.026504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.026743 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.027385 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:28.027391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:28.027699 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 14:58:28.027928 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 14:58:28.028385 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":329,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 14:58:28.031328 7696 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39595
I20260925 14:58:28.031610 7696 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:28.032362 7696 heartbeater.cc:511] Master 127.4.243.254:39595 requested a full tablet report, sending...
I20260925 14:58:28.034647 7507 ts_manager.cc:194] Registered new tserver with Master: 01bc173840a24e0699adf276edef1faf (127.4.243.193:38047)
I20260925 14:58:28.036188 7507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:49917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:28.054118 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39595
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:28.374140 7702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:28.374455 7702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:28.374634 7702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:28.381989 7702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:28.382925 7702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:28.383128 7702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:28.393458 7702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:28.394719 7702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:28.396291 7702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:28.410581 7710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.410620 7708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.410578 7707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:28.411511 7702 server_base.cc:1034] running on GCE node
I20260925 14:58:28.412153 7702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:28.413128 7702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:28.414346 7702 hybrid_clock.cc:648] HybridClock initialized: now 1790348308414298 us; error 57 us; skew 500 ppm
I20260925 14:58:28.414706 7702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:28 dist-test-slave-qxh0 krb5kdc[7462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:28.417783 7702 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:28.419284 7702 webserver.cc:460] Webserver started at http://127.4.243.194:34309/ using document root <none> and password file <none>
I20260925 14:58:28.419965 7702 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:28.420105 7702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:28.420397 7702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:28.423216 7702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "dd241b1468624defac7ae313e6660895"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-qxh0"
server_key: "f60070b24bfbbc7b3be87b39c347cc12"
server_key_iv: "50ecc9694cf5de35c036fd444a59a885"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.423988 7702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "dd241b1468624defac7ae313e6660895"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-qxh0"
server_key: "f60070b24bfbbc7b3be87b39c347cc12"
server_key_iv: "50ecc9694cf5de35c036fd444a59a885"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.490249 7702 fs_manager.cc:691] Time spent creating directory manager: real 0.066s user 0.006s sys 0.000s
I20260925 14:58:28.494005 7717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.495388 7702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:28.495607 7702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "dd241b1468624defac7ae313e6660895"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-qxh0"
server_key: "f60070b24bfbbc7b3be87b39c347cc12"
server_key_iv: "50ecc9694cf5de35c036fd444a59a885"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.495824 7702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:28.533320 7702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:28.538986 7702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:28.539278 7702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:28.540334 7702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:28.542258 7702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:28.542348 7702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.542470 7702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:28.542541 7702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.577162 7702 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:34155
I20260925 14:58:28.577209 7829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:34155 every 8 connection(s)
I20260925 14:58:28.578491 7702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:28.581913 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 7702
I20260925 14:58:28.582109 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:28.582548 5071 external_mini_cluster.cc:1428] Setting key dc2a5a9861d1965111c25113e96de638
Sep 25 14:58:28 dist-test-slave-qxh0 krb5kdc[7462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:28.600133 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.581044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:56321 (local address 127.4.243.254:39595)
0925 14:58:28.581624 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:28.581632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:28.582693 (+ 1061us) server_negotiation.cc:407] Connection header received
0925 14:58:28.584718 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:28.584723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:28.584799 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:28.584915 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:28.586340 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.587512 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.589297 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.589682 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.589819 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.594561 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:28.594566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:28.594569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:28.594647 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:28.597267 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.597941 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.597947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.597951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.598077 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.598478 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.598483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.598486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.598789 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:28.598931 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:28.599580 (+ 649us) server_negotiation.cc:299] Negotiation successful
0925 14:58:28.599790 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:28.601188 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.581629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39595 (local address 127.4.243.194:56321)
0925 14:58:28.582422 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:28.582485 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:28.584471 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:28.585104 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:28.585112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:28.585584 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:28.586174 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.586189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.587681 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.587686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.589152 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.589164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.589897 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.589904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.590119 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.591722 (+ 1603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:28.591758 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:28.594353 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:28.597410 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.597418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.597433 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.597824 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.598182 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.598187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.598191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.598362 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.598905 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:28.598913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:28.599242 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 14:58:28.599486 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 14:58:28.599999 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":344,"threads_started":1}
I20260925 14:58:28.602489 7830 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39595
I20260925 14:58:28.602837 7830 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:28.603585 7830 heartbeater.cc:511] Master 127.4.243.254:39595 requested a full tablet report, sending...
I20260925 14:58:28.605059 7507 ts_manager.cc:194] Registered new tserver with Master: dd241b1468624defac7ae313e6660895 (127.4.243.194:34155)
I20260925 14:58:28.605911 7507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:56321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 14:58:29.039784 7696 heartbeater.cc:503] Master 127.4.243.254:39595 was elected leader, sending a full tablet report...
I20260925 14:58:29.608239 7830 heartbeater.cc:503] Master 127.4.243.254:39595 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
W20260925 14:58:29.813957 5071 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.243.195: real 1.229s user 0.000s sys 0.009s
I20260925 14:58:29.814826 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39595
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:30.146230 7836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:30.146497 7836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:30.146607 7836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:30.153760 7836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:30.153872 7836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:30.153975 7836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:30.164044 7836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:30.165263 7836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:30.166563 7836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:30.179962 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.180006 7842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.180030 7844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:30.180624 7836 server_base.cc:1034] running on GCE node
I20260925 14:58:30.181298 7836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:30.182273 7836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:30.183529 7836 hybrid_clock.cc:648] HybridClock initialized: now 1790348310183495 us; error 49 us; skew 500 ppm
I20260925 14:58:30.183892 7836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:30 dist-test-slave-qxh0 krb5kdc[7462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:30.187165 7836 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:30.188740 7836 webserver.cc:460] Webserver started at http://127.4.243.195:42163/ using document root <none> and password file <none>
I20260925 14:58:30.189407 7836 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:30.189543 7836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:30.189855 7836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:30.192875 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "99513651eb8b43c79b113f00166c707a"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-qxh0"
server_key: "d2d49a6b0fa56ee127fe5b9dd630c18e"
server_key_iv: "0c6454eb828b4f552ece91348b4435f4"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.193683 7836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "99513651eb8b43c79b113f00166c707a"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-qxh0"
server_key: "d2d49a6b0fa56ee127fe5b9dd630c18e"
server_key_iv: "0c6454eb828b4f552ece91348b4435f4"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.199396 7836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 14:58:30.202826 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.204097 7836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:58:30.204260 7836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "99513651eb8b43c79b113f00166c707a"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-qxh0"
server_key: "d2d49a6b0fa56ee127fe5b9dd630c18e"
server_key_iv: "0c6454eb828b4f552ece91348b4435f4"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.204470 7836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:30.233587 7836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:30.237936 7836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:30.238231 7836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:30.239225 7836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:30.240538 7836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:30.240639 7836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.240797 7836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:30.240867 7836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.273368 7836 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:46231
I20260925 14:58:30.273412 7963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:46231 every 8 connection(s)
I20260925 14:58:30.274678 7836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:30.282130 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 7836
I20260925 14:58:30.282302 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:30.282786 5071 external_mini_cluster.cc:1428] Setting key f8feb041258f44cb0dd471b7fc1aeba4
Sep 25 14:58:30 dist-test-slave-qxh0 krb5kdc[7462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:30.294406 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.277138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:40993 (local address 127.4.243.254:39595)
0925 14:58:30.277578 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:30.277582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:30.278543 (+ 961us) server_negotiation.cc:407] Connection header received
0925 14:58:30.280430 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:30.280436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:30.280522 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:30.280640 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:30.282132 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.283109 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.284817 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.285352 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.285476 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.289139 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:30.289144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:30.289147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:30.289194 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:30.291918 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.292557 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.292561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.292563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.292666 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.293037 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.293040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.293042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.293285 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:30.293386 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:30.293989 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 14:58:30.294129 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260925 14:58:30.295351 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.277743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39595 (local address 127.4.243.195:40993)
0925 14:58:30.278366 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:30.278415 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:30.280159 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:30.280871 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:30.280879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:30.281422 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:30.281976 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.281992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.283269 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.283273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.284682 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.284693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.285585 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.285594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.285822 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.286531 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:30.286552 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:30.288952 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:30.292061 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.292067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.292078 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.292442 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.292759 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.292762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.292764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.292915 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.293381 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:30.293386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:30.293700 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 14:58:30.293919 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 14:58:30.294252 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":240,"threads_started":1}
I20260925 14:58:30.296802 7964 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39595
I20260925 14:58:30.297144 7964 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:30.297984 7964 heartbeater.cc:511] Master 127.4.243.254:39595 requested a full tablet report, sending...
I20260925 14:58:30.299718 7507 ts_manager.cc:194] Registered new tserver with Master: 99513651eb8b43c79b113f00166c707a (127.4.243.195:46231)
I20260925 14:58:30.300607 7507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:40993
I20260925 14:58:30.310349 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:58:30.339517 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:30.336875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.4.243.254:39595)
0925 14:58:30.337098 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:30.337108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:30.337536 (+ 428us) server_negotiation.cc:407] Connection header received
0925 14:58:30.337756 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:30.337762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:30.337847 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:30.337953 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:30.339216 (+ 1263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47068: BlockingRecv error: recv got EOF from 127.0.0.1:47068 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:58:30.342571 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7568
W20260925 14:58:30.345302 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7568: Not authorized: Client connection negotiation failed: client connection to 127.4.243.193:38047: server requires authentication, but client does not have Kerberos credentials available
I20260925 14:58:30.345487 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 7568
I20260925 14:58:30.360484 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7702
W20260925 14:58:30.363112 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7702: Not authorized: Client connection negotiation failed: client connection to 127.4.243.194:34155: server requires authentication, but client does not have Kerberos credentials available
I20260925 14:58:30.363247 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 7702
I20260925 14:58:30.380041 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7836
W20260925 14:58:30.382923 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7836: Not authorized: Client connection negotiation failed: client connection to 127.4.243.195:46231: server requires authentication, but client does not have Kerberos credentials available
I20260925 14:58:30.383118 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 7836
I20260925 14:58:30.398231 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7475
I20260925 14:58:30.399375 7537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:30.503082 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 7475
2026-09-25T14:58:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4761 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:32 dist-test-slave-qxh0 krb5kdc[7982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:32 dist-test-slave-qxh0 krb5kdc[7982](info): set up 2 sockets
Sep 25 14:58:32 dist-test-slave-qxh0 krb5kdc[7982](info): commencing operation
krb5kdc: starting...
W20260925 14:58:33.274662 5071 mini_kdc.cc:121] Time spent starting KDC: real 2.731s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:33 dist-test-slave-qxh0 krb5kdc[7982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:33.453903 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35579
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:33.761411 7995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:33.761667 7995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:33.761823 7995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:33.768464 7995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:33.768559 7995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:33.768661 7995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:33.768754 7995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:33.768844 7995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:33.777832 7995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:33.779016 7995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:33.780217 7995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:33.791570 8001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:33.791692 8000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:33.791743 8003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:33.792639 7995 server_base.cc:1034] running on GCE node
I20260925 14:58:33.793320 7995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:33.794670 7995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:33.795883 7995 hybrid_clock.cc:648] HybridClock initialized: now 1790348313795838 us; error 54 us; skew 500 ppm
I20260925 14:58:33.796212 7995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:33 dist-test-slave-qxh0 krb5kdc[7982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:33.799358 7995 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:33.800859 7995 webserver.cc:460] Webserver started at http://127.4.243.254:42571/ using document root <none> and password file <none>
I20260925 14:58:33.801512 7995 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:33.801651 7995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:33.801951 7995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:33.804811 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "ee31acc5f931499e87bc0b4433448183"
format_stamp: "Formatted at 2026-09-25 14:58:33 on dist-test-slave-qxh0"
server_key: "3eb6a010bb82437419a96d8f24dd2ac9"
server_key_iv: "89377d8f0102a2e56daa3544656a8e22"
server_key_version: "encryptionkey@0"
I20260925 14:58:33.805588 7995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "ee31acc5f931499e87bc0b4433448183"
format_stamp: "Formatted at 2026-09-25 14:58:33 on dist-test-slave-qxh0"
server_key: "3eb6a010bb82437419a96d8f24dd2ac9"
server_key_iv: "89377d8f0102a2e56daa3544656a8e22"
server_key_version: "encryptionkey@0"
I20260925 14:58:33.810496 7995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 14:58:33.813802 8010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:33.815102 7995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:58:33.815284 7995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "ee31acc5f931499e87bc0b4433448183"
format_stamp: "Formatted at 2026-09-25 14:58:33 on dist-test-slave-qxh0"
server_key: "3eb6a010bb82437419a96d8f24dd2ac9"
server_key_iv: "89377d8f0102a2e56daa3544656a8e22"
server_key_version: "encryptionkey@0"
I20260925 14:58:33.815507 7995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:33.834219 7995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:33.838280 7995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:33.838577 7995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:33.856639 7995 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:35579
I20260925 14:58:33.856660 8061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:35579 every 8 connection(s)
I20260925 14:58:33.857933 7995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:33.859953 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 7995
I20260925 14:58:33.860179 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:33.861016 5071 external_mini_cluster.cc:1428] Setting key 149c8a3a91a8695e338347a50ef700e3
I20260925 14:58:33.862313 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:58:33 dist-test-slave-qxh0 krb5kdc[7982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:33.875957 8062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183: Bootstrap starting.
I20260925 14:58:33.878939 8062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:33.879876 8062 log.cc:824] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:33.882388 8062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183: No bootstrap required, opened a new log
I20260925 14:58:33.882718 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.862901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.4.243.254:35579)
0925 14:58:33.863669 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:33.863680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:33.863719 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:58:33.865165 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:33.865184 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:33.865581 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:33.865915 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:33.867779 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.869118 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.870809 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.871439 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.871565 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:33.875990 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:33.875998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:33.876013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:33.876045 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:33.878978 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:33.879669 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:33.879675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:33.879681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:33.879836 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:33.880217 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:33.880221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:33.880222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:33.880710 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:33.881054 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:33.881372 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 14:58:33.881739 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":359,"threads_started":1}
I20260925 14:58:33.886255 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:33.889736 8062 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee31acc5f931499e87bc0b4433448183" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35579 } }
I20260925 14:58:33.890086 8062 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:33.890188 8062 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee31acc5f931499e87bc0b4433448183, State: Initialized, Role: FOLLOWER
I20260925 14:58:33.890769 8062 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee31acc5f931499e87bc0b4433448183" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35579 } }
I20260925 14:58:33.891012 8062 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:33.891139 8062 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:33.891302 8062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:33.893682 8062 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee31acc5f931499e87bc0b4433448183" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35579 } }
I20260925 14:58:33.894237 8062 leader_election.cc:302] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee31acc5f931499e87bc0b4433448183; no voters:
I20260925 14:58:33.894865 8062 leader_election.cc:288] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:33.894982 8068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:33.896387 8068 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [term 1 LEADER]: Becoming Leader. State: Replica: ee31acc5f931499e87bc0b4433448183, State: Running, Role: LEADER
I20260925 14:58:33.896974 8062 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:33.896889 8068 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee31acc5f931499e87bc0b4433448183" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35579 } }
I20260925 14:58:33.899667 8070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee31acc5f931499e87bc0b4433448183. Latest consensus state: current_term: 1 leader_uuid: "ee31acc5f931499e87bc0b4433448183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee31acc5f931499e87bc0b4433448183" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35579 } } }
I20260925 14:58:33.899708 8069 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee31acc5f931499e87bc0b4433448183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee31acc5f931499e87bc0b4433448183" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35579 } } }
I20260925 14:58:33.900048 8070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:33.900048 8069 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:33.904404 8073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:33.910506 8073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:33.940860 8073 catalog_manager.cc:1347] Generated new cluster ID: 9d96023af77a4e4994869a069113810f
I20260925 14:58:33.940994 8073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:33.957436 8090 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:58:33.969190 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.960162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.4.243.254:35579)
0925 14:58:33.960350 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:33.960356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:33.960956 (+ 600us) server_negotiation.cc:407] Connection header received
0925 14:58:33.961142 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:33.961148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:33.961227 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:33.961327 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:33.962529 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.963635 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.965444 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.966064 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.966191 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:33.967446 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:33.967452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:33.967457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:33.968263 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:33.968317 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:33.968347 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:33.968455 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:33.968613 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:58:33.968699 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 14:58:33.994748 8073 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:33.996047 8073 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:34.004149 8073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee31acc5f931499e87bc0b4433448183: Generated new TSK 0
I20260925 14:58:34.005105 8073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:34.016259 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 7995
I20260925 14:58:34.017684 8057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:34.108649 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 7995
2026-09-25T14:58:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3600 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:34 dist-test-slave-qxh0 krb5kdc[8096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:34 dist-test-slave-qxh0 krb5kdc[8096](info): set up 2 sockets
Sep 25 14:58:34 dist-test-slave-qxh0 krb5kdc[8096](info): commencing operation
krb5kdc: starting...
W20260925 14:58:35.166904 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:35 dist-test-slave-qxh0 krb5kdc[8096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:35.346493 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43131
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:43131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:35.656790 8109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:35.657033 8109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:35.657133 8109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:35.663928 8109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:35.664018 8109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:35.664062 8109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:35.664134 8109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:35.664214 8109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:35.673617 8109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:43131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:35.674885 8109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:35.676191 8109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:35.687412 8115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:35.687423 8114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:35.687460 8117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:35.688059 8109 server_base.cc:1034] running on GCE node
I20260925 14:58:35.688784 8109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:35.690037 8109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:35.691243 8109 hybrid_clock.cc:648] HybridClock initialized: now 1790348315691209 us; error 50 us; skew 500 ppm
I20260925 14:58:35.691558 8109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:35 dist-test-slave-qxh0 krb5kdc[8096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:35.694710 8109 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:35.697489 8109 webserver.cc:460] Webserver started at http://127.4.243.254:38103/ using document root <none> and password file <none>
I20260925 14:58:35.698177 8109 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:35.698309 8109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:35.698644 8109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:35.701417 8109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe"
format_stamp: "Formatted at 2026-09-25 14:58:35 on dist-test-slave-qxh0"
server_key: "29aac5972fc51a9ff0328cdc187752a5"
server_key_iv: "894dbe4f208ba91ec5d64cc31d8e7b23"
server_key_version: "encryptionkey@0"
I20260925 14:58:35.702165 8109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe"
format_stamp: "Formatted at 2026-09-25 14:58:35 on dist-test-slave-qxh0"
server_key: "29aac5972fc51a9ff0328cdc187752a5"
server_key_iv: "894dbe4f208ba91ec5d64cc31d8e7b23"
server_key_version: "encryptionkey@0"
I20260925 14:58:35.706996 8109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 14:58:35.710152 8124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:35.711364 8109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:35.711557 8109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe"
format_stamp: "Formatted at 2026-09-25 14:58:35 on dist-test-slave-qxh0"
server_key: "29aac5972fc51a9ff0328cdc187752a5"
server_key_iv: "894dbe4f208ba91ec5d64cc31d8e7b23"
server_key_version: "encryptionkey@0"
I20260925 14:58:35.711732 8109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:35.729568 8109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:35.733484 8109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:35.733747 8109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:35.750771 8109 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:43131
I20260925 14:58:35.750810 8175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:43131 every 8 connection(s)
I20260925 14:58:35.752036 8109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:35.753176 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8109
I20260925 14:58:35.753403 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:35.754205 5071 external_mini_cluster.cc:1428] Setting key 0380efbd05ef30b5da18a6f6325d788f
I20260925 14:58:35.756480 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:58:35 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:35.768953 8176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe: Bootstrap starting.
I20260925 14:58:35.772909 8176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:35.774160 8176 log.cc:824] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:35.776384 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.756145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.4.243.254:43131)
0925 14:58:35.756848 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:35.756859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:35.756897 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:58:35.758357 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:35.758375 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:35.758794 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:35.759097 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:35.760698 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.761997 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.763746 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.764348 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.764471 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.768812 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:35.768822 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:35.768839 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:35.768884 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:35.772551 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:35.773160 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:35.773169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:35.773176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:35.773395 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:35.773736 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:35.773740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:35.773744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:35.774371 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:35.774745 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:35.775067 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 14:58:35.775412 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":349,"threads_started":1}
I20260925 14:58:35.777050 8176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe: No bootstrap required, opened a new log
I20260925 14:58:35.784411 8176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43131 } }
I20260925 14:58:35.784739 8176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:35.784834 8176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afca88e1ce44c44baa3c0ba5a3a5ffe, State: Initialized, Role: FOLLOWER
I20260925 14:58:35.785343 8176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43131 } }
I20260925 14:58:35.785565 8176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:35.785696 8176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:35.785861 8176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:35.787890 8176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43131 } }
I20260925 14:58:35.788317 8176 leader_election.cc:302] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3afca88e1ce44c44baa3c0ba5a3a5ffe; no voters:
I20260925 14:58:35.788864 8176 leader_election.cc:288] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:35.788977 8181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:35.790122 8181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [term 1 LEADER]: Becoming Leader. State: Replica: 3afca88e1ce44c44baa3c0ba5a3a5ffe, State: Running, Role: LEADER
I20260925 14:58:35.790745 8181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43131 } }
I20260925 14:58:35.790848 8176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:35.793442 8183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3afca88e1ce44c44baa3c0ba5a3a5ffe. Latest consensus state: current_term: 1 leader_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43131 } } }
I20260925 14:58:35.793769 8183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:35.794243 8182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afca88e1ce44c44baa3c0ba5a3a5ffe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43131 } } }
I20260925 14:58:35.794556 8182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:35.794709 8190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:35.799621 8190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:35.807317 8190 catalog_manager.cc:1347] Generated new cluster ID: 6024afe2ca6547e7824d1bdb97f72174
I20260925 14:58:35.807435 8190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:35.819164 8190 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:35.820387 8190 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:35.833750 8190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3afca88e1ce44c44baa3c0ba5a3a5ffe: Generated new TSK 0
I20260925 14:58:35.834564 8190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:35.884029 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43131
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:36.245760 8202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:36.246017 8202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:36.246109 8202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:36.252939 8202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:36.253036 8202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:36.253141 8202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:36.262163 8202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:36.263327 8202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:36.264473 8202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:36.277282 8207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:36.277307 8210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:36.277267 8208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:36.278700 8202 server_base.cc:1034] running on GCE node
I20260925 14:58:36.279304 8202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:36.280182 8202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:36.281370 8202 hybrid_clock.cc:648] HybridClock initialized: now 1790348316281324 us; error 59 us; skew 500 ppm
I20260925 14:58:36.281666 8202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:36 dist-test-slave-qxh0 krb5kdc[8096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:36.284659 8202 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:36.285944 8202 webserver.cc:460] Webserver started at http://127.4.243.193:46043/ using document root <none> and password file <none>
I20260925 14:58:36.286638 8202 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:36.286782 8202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:36.287075 8202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:36.289769 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "5ba7165218084bcda9fc176ca8ef83a1"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-qxh0"
server_key: "06a54d719719fbebd1f67f6ba497ed76"
server_key_iv: "5481a1fc72791fc65008506a7367a1e8"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.290555 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "5ba7165218084bcda9fc176ca8ef83a1"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-qxh0"
server_key: "06a54d719719fbebd1f67f6ba497ed76"
server_key_iv: "5481a1fc72791fc65008506a7367a1e8"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.295549 8202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 14:58:36.298857 8217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.300010 8202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 14:58:36.300168 8202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "5ba7165218084bcda9fc176ca8ef83a1"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-qxh0"
server_key: "06a54d719719fbebd1f67f6ba497ed76"
server_key_iv: "5481a1fc72791fc65008506a7367a1e8"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.300369 8202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:36.311189 8202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:36.314926 8202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:36.315178 8202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:36.315948 8202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:36.317191 8202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:36.317274 8202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.317422 8202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:36.317500 8202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.349287 8202 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:34189
I20260925 14:58:36.349357 8329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:34189 every 8 connection(s)
I20260925 14:58:36.350502 8202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:36.352505 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8202
I20260925 14:58:36.352730 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:36.353576 5071 external_mini_cluster.cc:1428] Setting key 2c8f675bbd33d1c1fbdc55418ebdc75c
Sep 25 14:58:36 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:36.373394 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:36.353158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:37319 (local address 127.4.243.254:43131)
0925 14:58:36.353717 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:36.353724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:36.354921 (+ 1197us) server_negotiation.cc:407] Connection header received
0925 14:58:36.357375 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:36.357379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:36.357440 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:36.357528 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:36.359584 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.360273 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.362198 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.362961 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.363105 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:36.366567 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:36.366575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:36.366583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:36.366649 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:36.370255 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:36.371011 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:36.371018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:36.371022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:36.371159 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:36.371562 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:36.371568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:36.371572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:36.371946 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:36.372104 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:36.372742 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 14:58:36.372992 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":322,"threads_started":1}
I20260925 14:58:36.374012 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:36.353872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43131 (local address 127.4.243.193:37319)
0925 14:58:36.354765 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:36.354808 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:36.357076 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:36.357800 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:36.357814 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:36.358567 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:36.359390 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:36.359418 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.360431 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.360439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:36.362052 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:36.362067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.363163 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.363167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:36.363341 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:36.364037 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:36.364057 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:36.366349 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:36.370405 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:36.370413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:36.370426 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:36.370901 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:36.371266 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:36.371269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:36.371271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:36.371431 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:36.372069 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:36.372074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:36.372356 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 14:58:36.372563 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 14:58:36.372917 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":380,"threads_started":1}
I20260925 14:58:36.375682 8330 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 14:58:36.376097 8330 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:36.376776 8330 heartbeater.cc:511] Master 127.4.243.254:43131 requested a full tablet report, sending...
I20260925 14:58:36.379047 8141 ts_manager.cc:194] Registered new tserver with Master: 5ba7165218084bcda9fc176ca8ef83a1 (127.4.243.193:34189)
I20260925 14:58:36.381033 8141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:37319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:36.400064 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43131
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:36.705658 8336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:36.705941 8336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:36.706104 8336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:36.713058 8336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:36.713194 8336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:36.713382 8336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:36.722978 8336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:36.724213 8336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:36.725549 8336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:36.737726 8341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:36.737757 8342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:36.737977 8344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:36.738847 8336 server_base.cc:1034] running on GCE node
I20260925 14:58:36.739315 8336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:36.740092 8336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:36.741271 8336 hybrid_clock.cc:648] HybridClock initialized: now 1790348316741232 us; error 27 us; skew 500 ppm
I20260925 14:58:36.741528 8336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:36 dist-test-slave-qxh0 krb5kdc[8096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:36.748576 8336 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:36.750022 8336 webserver.cc:460] Webserver started at http://127.4.243.194:45477/ using document root <none> and password file <none>
I20260925 14:58:36.750773 8336 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:36.750895 8336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:36.751194 8336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:36.753883 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "f6cc27f5f3f24c44a51f0123e2931813"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-qxh0"
server_key: "6690acfaae19c61699a7e0ba23784dde"
server_key_iv: "425b215255d146a3760b152a424a49a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.754665 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "f6cc27f5f3f24c44a51f0123e2931813"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-qxh0"
server_key: "6690acfaae19c61699a7e0ba23784dde"
server_key_iv: "425b215255d146a3760b152a424a49a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.759502 8336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:58:36.762688 8351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.763967 8336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:58:36.764122 8336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "f6cc27f5f3f24c44a51f0123e2931813"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-qxh0"
server_key: "6690acfaae19c61699a7e0ba23784dde"
server_key_iv: "425b215255d146a3760b152a424a49a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.764292 8336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:36.783915 8336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:36.787411 8336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:36.787690 8336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:36.788532 8336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:36.789731 8336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:36.789831 8336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.789966 8336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:36.790035 8336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.822775 8336 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:46277
I20260925 14:58:36.822835 8463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:46277 every 8 connection(s)
I20260925 14:58:36.823997 8336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:36.826095 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8336
I20260925 14:58:36.826337 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:36.826889 5071 external_mini_cluster.cc:1428] Setting key 4cba86d08433ec3cb38dca90095267f4
Sep 25 14:58:36 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:36.844551 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:36.826430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:48797 (local address 127.4.243.254:43131)
0925 14:58:36.826648 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:36.826655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:36.828006 (+ 1351us) server_negotiation.cc:407] Connection header received
0925 14:58:36.830529 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:36.830533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:36.830663 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:36.830761 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:36.832847 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.833558 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.835253 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.835666 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.835811 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:36.839164 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:36.839169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:36.839172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:36.839200 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:36.841586 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:36.842276 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:36.842294 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:36.842302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:36.842441 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:36.842947 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:36.842950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:36.842952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:36.843194 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:36.843343 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:36.843998 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 14:58:36.844205 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:36.845507 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:36.827067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43131 (local address 127.4.243.194:48797)
0925 14:58:36.827861 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:36.827908 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:36.830180 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:36.831032 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:36.831065 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:36.831865 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:36.832664 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:36.832692 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.833742 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.833746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:36.835098 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:36.835110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.835896 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.835903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:36.836118 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:36.836817 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:36.836839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:36.838952 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:36.841725 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:36.841734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:36.841749 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:36.842149 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:36.842561 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:36.842567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:36.842571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:36.842777 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:36.843332 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:36.843339 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:36.843638 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 14:58:36.843884 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 14:58:36.844416 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260925 14:58:36.846925 8464 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43131
I20260925 14:58:36.847249 8464 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 14:58:36.848129 8464 heartbeater.cc:511] Master 127.4.243.254:43131 requested a full tablet report, sending...
I20260925 14:58:36.849671 8141 ts_manager.cc:194] Registered new tserver with Master: f6cc27f5f3f24c44a51f0123e2931813 (127.4.243.194:46277)
I20260925 14:58:36.850486 8141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:48797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:36.870370 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43131
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:37.178227 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:37.178596 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:37.178787 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:37.185777 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:37.185935 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:37.186122 8469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:37.195923 8469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:37.197255 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:37.198554 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:37.211195 8477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:37.211324 8474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:37.211267 8475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:37.212435 8469 server_base.cc:1034] running on GCE node
I20260925 14:58:37.212945 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:37.213771 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:37.215189 8469 hybrid_clock.cc:648] HybridClock initialized: now 1790348317215121 us; error 33 us; skew 500 ppm
I20260925 14:58:37.215528 8469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:37.218528 8469 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:37.220026 8469 webserver.cc:460] Webserver started at http://127.4.243.195:34515/ using document root <none> and password file <none>
I20260925 14:58:37.220734 8469 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:37.220857 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:37.221148 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:37.223884 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "5a1a861075844698ac64ce52bea834e4"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-qxh0"
server_key: "4abf0b2a1c9db76d393967b129af4767"
server_key_iv: "c76a87c79f33f2b2a25874541b63e6d1"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.224643 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "5a1a861075844698ac64ce52bea834e4"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-qxh0"
server_key: "4abf0b2a1c9db76d393967b129af4767"
server_key_iv: "c76a87c79f33f2b2a25874541b63e6d1"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.229504 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 14:58:37.232717 8484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.233953 8469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:58:37.234123 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "5a1a861075844698ac64ce52bea834e4"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-qxh0"
server_key: "4abf0b2a1c9db76d393967b129af4767"
server_key_iv: "c76a87c79f33f2b2a25874541b63e6d1"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.234329 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:37.260473 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:37.264439 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:37.264717 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:37.265622 8469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:37.266922 8469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:37.267001 8469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.267167 8469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:37.267215 8469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.299608 8469 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:34547
I20260925 14:58:37.299655 8596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:34547 every 8 connection(s)
I20260925 14:58:37.300805 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:37.305963 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8469
I20260925 14:58:37.306150 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:37.306576 5071 external_mini_cluster.cc:1428] Setting key 6095210036b79d4713134d9b03856d4d
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:37.320742 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.303216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:39703 (local address 127.4.243.254:43131)
0925 14:58:37.303410 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.303416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.304681 (+ 1265us) server_negotiation.cc:407] Connection header received
0925 14:58:37.306528 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.306534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.306670 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.306787 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.308606 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.309770 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.311388 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.311822 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.311935 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.315452 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.315456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.315460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.315487 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.317995 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.318699 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.318702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.318704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.318820 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.319200 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.319203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.319205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.319446 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.319547 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.320184 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.320390 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:58:37.321861 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.303745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43131 (local address 127.4.243.195:39703)
0925 14:58:37.304482 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.304521 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.306275 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.307077 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.307089 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.307604 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.308416 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.308449 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.309928 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.309933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.311240 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.311255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.312109 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.312115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.312318 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.312923 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.312944 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.315214 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.318109 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.318115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.318127 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.318537 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.318911 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.318915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.318917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.319076 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.319551 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.319556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.319822 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.320043 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.320495 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":313,"threads_started":1}
I20260925 14:58:37.323107 8597 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43131
I20260925 14:58:37.323374 8597 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:37.323997 8597 heartbeater.cc:511] Master 127.4.243.254:43131 requested a full tablet report, sending...
I20260925 14:58:37.325265 8141 ts_manager.cc:194] Registered new tserver with Master: 5a1a861075844698ac64ce52bea834e4 (127.4.243.195:34547)
I20260925 14:58:37.326121 8141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:39703
I20260925 14:58:37.334151 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:37.380709 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.368090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.4.243.254:43131)
0925 14:58:37.368266 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.368271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.368760 (+ 489us) server_negotiation.cc:407] Connection header received
0925 14:58:37.368906 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.368910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.368973 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.369064 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.370359 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.371112 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.372269 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.372634 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.372723 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.376411 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.376416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.376419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.376448 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.378767 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.379305 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.379308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.379310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.379418 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.379697 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.379699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.379701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.379981 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.380147 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.380362 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.380500 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 14:58:37.381228 8141 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43832
I20260925 14:58:37.383018 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8202
I20260925 14:58:37.383893 8330 heartbeater.cc:503] Master 127.4.243.254:43131 was elected leader, sending a full tablet report...
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:37.398108 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.383936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.4.243.193:34189)
0925 14:58:37.384528 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.384537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.384563 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:58:37.384618 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.384625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.384880 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.385033 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.386311 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.387381 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.388620 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.389129 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.389227 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.392938 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.392948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.392962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.392993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.395827 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.396416 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.396422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.396424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.396535 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.396852 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.396855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.396857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.397201 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.397414 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.397667 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.397840 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":293,"threads_started":1}
W20260925 14:58:37.399366 8325 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34002
W20260925 14:58:37.400327 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8202: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:37.400466 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8202
I20260925 14:58:37.416055 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8336
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:37.431360 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.416821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.4.243.194:46277)
0925 14:58:37.417384 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.417396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.417419 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:58:37.417495 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.417500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.417852 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.418016 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.419204 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.420140 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.421419 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.421968 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.422069 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.425844 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.425855 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.425869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.425905 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.428723 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.429344 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.429351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.429353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.429471 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.429820 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.429823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.429825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.430146 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.430409 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.430768 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.431004 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":319,"threads_started":1}
W20260925 14:58:37.432634 8459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53578
W20260925 14:58:37.433683 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8336: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:37.433863 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8336
I20260925 14:58:37.449347 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8469
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:37.465574 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.450297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.4.243.195:34547)
0925 14:58:37.451049 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.451056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.451085 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:58:37.451139 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.451145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.451362 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.451541 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.452833 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.453848 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.455160 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.455707 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.455862 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.460232 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.460243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.460257 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.460287 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.463237 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.463791 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.463797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.463799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.463904 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.464188 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.464191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.464193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.464508 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.464718 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.464997 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.465176 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":476,"threads_started":1}
W20260925 14:58:37.466845 8592 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49802
W20260925 14:58:37.467967 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 14:58:37.468130 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8469
I20260925 14:58:37.483136 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8109
I20260925 14:58:37.484149 8171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:37.582573 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8109
2026-09-25T14:58:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3479 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8614](info): set up 2 sockets
Sep 25 14:58:37 dist-test-slave-qxh0 krb5kdc[8614](info): commencing operation
krb5kdc: starting...
W20260925 14:58:38.658856 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:38 dist-test-slave-qxh0 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:38.841526 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37773
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:36763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:37773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:39.147317 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:39.147570 8627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:39.147650 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:39.154443 8627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:39.154539 8627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:39.154582 8627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:39.154703 8627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:39.154748 8627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:39.164016 8627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:36763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:37773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:39.165200 8627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:39.166488 8627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:39.177778 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:39.177786 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:39.177634 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:39.178882 8627 server_base.cc:1034] running on GCE node
I20260925 14:58:39.179610 8627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:39.180938 8627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:39.182164 8627 hybrid_clock.cc:648] HybridClock initialized: now 1790348319182102 us; error 53 us; skew 500 ppm
I20260925 14:58:39.182528 8627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:39 dist-test-slave-qxh0 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:39.185937 8627 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:39.187565 8627 webserver.cc:460] Webserver started at http://127.4.243.254:34229/ using document root <none> and password file <none>
I20260925 14:58:39.188254 8627 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:39.188398 8627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:39.188717 8627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:39.191663 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c"
format_stamp: "Formatted at 2026-09-25 14:58:39 on dist-test-slave-qxh0"
server_key: "4fbc171d56e1949f70c33fbc6cf65580"
server_key_iv: "3b4bb627cfd7403088160b958104112a"
server_key_version: "encryptionkey@0"
I20260925 14:58:39.192435 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c"
format_stamp: "Formatted at 2026-09-25 14:58:39 on dist-test-slave-qxh0"
server_key: "4fbc171d56e1949f70c33fbc6cf65580"
server_key_iv: "3b4bb627cfd7403088160b958104112a"
server_key_version: "encryptionkey@0"
I20260925 14:58:39.197551 8627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 14:58:39.200984 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:39.202289 8627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:58:39.202471 8627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c"
format_stamp: "Formatted at 2026-09-25 14:58:39 on dist-test-slave-qxh0"
server_key: "4fbc171d56e1949f70c33fbc6cf65580"
server_key_iv: "3b4bb627cfd7403088160b958104112a"
server_key_version: "encryptionkey@0"
I20260925 14:58:39.202736 8627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:39.223716 8627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:39.227737 8627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:39.228001 8627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:39.245143 8627 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:37773
I20260925 14:58:39.245169 8693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:37773 every 8 connection(s)
I20260925 14:58:39.246466 8627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:39.247777 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8627
I20260925 14:58:39.248013 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:39.248734 5071 external_mini_cluster.cc:1428] Setting key 65963d377ccbbeb55ae9159646dc7faa
I20260925 14:58:39.251081 8694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 14:58:39 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:39.264038 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c: Bootstrap starting.
I20260925 14:58:39.268139 8694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:39.269475 8694 log.cc:824] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:39.271611 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:39.250679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.4.243.254:37773)
0925 14:58:39.251462 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:39.251474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:39.251513 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:58:39.253118 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:39.253137 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:39.253528 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:39.253864 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:39.255666 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:39.256967 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:39.258692 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:39.259333 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:39.259452 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:39.263971 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:39.263981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:39.263999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:39.264052 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:39.267753 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:39.268420 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:39.268428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:39.268436 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:39.268671 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:39.268999 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:39.269004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:39.269007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:39.269683 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:39.269974 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:39.270310 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 14:58:39.270691 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":392,"threads_started":1}
I20260925 14:58:39.272362 8694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c: No bootstrap required, opened a new log
I20260925 14:58:39.279944 8694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37773 } }
I20260925 14:58:39.280277 8694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:39.280377 8694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8bef6aa25c44cfaeda7e3427d1fa3c, State: Initialized, Role: FOLLOWER
I20260925 14:58:39.280864 8694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37773 } }
I20260925 14:58:39.281080 8694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:39.281219 8694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:39.281380 8694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:39.283437 8694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37773 } }
I20260925 14:58:39.283883 8694 leader_election.cc:302] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c8bef6aa25c44cfaeda7e3427d1fa3c; no voters:
I20260925 14:58:39.284418 8694 leader_election.cc:288] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:39.284548 8699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:39.285158 8699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [term 1 LEADER]: Becoming Leader. State: Replica: 7c8bef6aa25c44cfaeda7e3427d1fa3c, State: Running, Role: LEADER
I20260925 14:58:39.285710 8699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37773 } }
I20260925 14:58:39.286289 8694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:39.288771 8701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c8bef6aa25c44cfaeda7e3427d1fa3c. Latest consensus state: current_term: 1 leader_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37773 } } }
I20260925 14:58:39.289158 8701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:39.289963 8708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:39.290205 8700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8bef6aa25c44cfaeda7e3427d1fa3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37773 } } }
I20260925 14:58:39.290464 8700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:39.294898 8708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:39.302384 8708 catalog_manager.cc:1347] Generated new cluster ID: 6f6b322308d2479fad7f51ee389f9563
I20260925 14:58:39.302479 8708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:39.324033 8708 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:39.325884 8708 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:39.338215 8708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c8bef6aa25c44cfaeda7e3427d1fa3c: Generated new TSK 0
I20260925 14:58:39.339275 8708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:39.388231 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37773
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:36763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:39.698956 8720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:39.699198 8720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:39.699349 8720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:39.707060 8720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:39.707155 8720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:39.707358 8720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:39.717758 8720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:36763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:39.719205 8720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:39.720453 8720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:39.733310 8725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:39.733408 8726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:39.733320 8728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:39.734404 8720 server_base.cc:1034] running on GCE node
I20260925 14:58:39.735033 8720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:39.735980 8720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:39.737311 8720 hybrid_clock.cc:648] HybridClock initialized: now 1790348319737251 us; error 63 us; skew 500 ppm
I20260925 14:58:39.737795 8720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:39 dist-test-slave-qxh0 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:39.741705 8720 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:39.743820 8720 webserver.cc:460] Webserver started at http://127.4.243.193:44315/ using document root <none> and password file <none>
I20260925 14:58:39.744617 8720 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:39.744769 8720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:39.745126 8720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:39.748104 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "3d5a9481009e4f03bb2fe2e5c338d7fe"
format_stamp: "Formatted at 2026-09-25 14:58:39 on dist-test-slave-qxh0"
server_key: "5e999890057eee7b1af0072d95597876"
server_key_iv: "50e486e7b090742571ef1e620fa53f83"
server_key_version: "encryptionkey@0"
I20260925 14:58:39.748857 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "3d5a9481009e4f03bb2fe2e5c338d7fe"
format_stamp: "Formatted at 2026-09-25 14:58:39 on dist-test-slave-qxh0"
server_key: "5e999890057eee7b1af0072d95597876"
server_key_iv: "50e486e7b090742571ef1e620fa53f83"
server_key_version: "encryptionkey@0"
I20260925 14:58:39.753940 8720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 14:58:39.757323 8735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:39.758474 8720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:39.758658 8720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "3d5a9481009e4f03bb2fe2e5c338d7fe"
format_stamp: "Formatted at 2026-09-25 14:58:39 on dist-test-slave-qxh0"
server_key: "5e999890057eee7b1af0072d95597876"
server_key_iv: "50e486e7b090742571ef1e620fa53f83"
server_key_version: "encryptionkey@0"
I20260925 14:58:39.758852 8720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:39.772161 8720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:39.776525 8720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:39.776816 8720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:39.777678 8720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:39.778951 8720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:39.779049 8720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:39.779189 8720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:39.779265 8720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:39.811812 8720 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:39753
I20260925 14:58:39.811857 8847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:39753 every 8 connection(s)
I20260925 14:58:39.813086 8720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:39.817080 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8720
I20260925 14:58:39.817276 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:39.817960 5071 external_mini_cluster.cc:1428] Setting key 74b3b2ba2f54c45130da2d07bf73525c
Sep 25 14:58:39 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:39.833129 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:39.815657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:39397 (local address 127.4.243.254:37773)
0925 14:58:39.816121 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:39.816128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:39.817156 (+ 1028us) server_negotiation.cc:407] Connection header received
0925 14:58:39.819225 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:39.819231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:39.819313 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:39.819432 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:39.820873 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:39.821573 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:39.823160 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:39.823572 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:39.823709 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:39.827615 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:39.827620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:39.827624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:39.827665 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:39.830122 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:39.830892 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:39.830897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:39.830902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:39.831013 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:39.831417 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:39.831422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:39.831425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:39.831681 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:39.831858 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:39.832524 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 14:58:39.832740 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":210,"threads_started":1}
I20260925 14:58:39.834067 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:39.816310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37773 (local address 127.4.243.193:39397)
0925 14:58:39.817004 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:39.817042 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:39.818934 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:39.819671 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:39.819680 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:39.820164 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:39.820712 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:39.820730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:39.821731 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:39.821736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:39.823022 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:39.823033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:39.823799 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:39.823809 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:39.824039 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:39.824708 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:39.824765 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:39.827396 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:39.830284 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:39.830292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:39.830308 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:39.830775 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:39.831118 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:39.831124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:39.831127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:39.831305 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:39.831831 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:39.831839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:39.832178 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 14:58:39.832434 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 14:58:39.832970 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260925 14:58:39.835759 8848 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37773
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:39.836084 8848 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:39.836789 8848 heartbeater.cc:511] Master 127.4.243.254:37773 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 14:58:39.839102 8659 ts_manager.cc:194] Registered new tserver with Master: 3d5a9481009e4f03bb2fe2e5c338d7fe (127.4.243.193:39753)
I20260925 14:58:39.840492 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:39397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:39.859733 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37773
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:36763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:40.178977 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:40.179245 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:40.179344 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:40.186578 8854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:40.186733 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:40.186857 8854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:40.196810 8854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:36763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:40.197999 8854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:40.199293 8854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:40.212333 8862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.212512 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.212438 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:40.213472 8854 server_base.cc:1034] running on GCE node
I20260925 14:58:40.214087 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:40.214986 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:40.216216 8854 hybrid_clock.cc:648] HybridClock initialized: now 1790348320216172 us; error 60 us; skew 500 ppm
I20260925 14:58:40.216516 8854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:40.219673 8854 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:40.221068 8854 webserver.cc:460] Webserver started at http://127.4.243.194:37237/ using document root <none> and password file <none>
I20260925 14:58:40.221716 8854 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:40.221868 8854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:40.222158 8854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:40.225046 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "734b31813a9f405dbcdf21711206e037"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-qxh0"
server_key: "ad4eaebc19f780fca70012eb3ccd5698"
server_key_iv: "7947d0cba9fab162779dfa0a3064f9eb"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.225847 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "734b31813a9f405dbcdf21711206e037"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-qxh0"
server_key: "ad4eaebc19f780fca70012eb3ccd5698"
server_key_iv: "7947d0cba9fab162779dfa0a3064f9eb"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.230882 8854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 14:58:40.234457 8869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.235836 8854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:40.236016 8854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "734b31813a9f405dbcdf21711206e037"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-qxh0"
server_key: "ad4eaebc19f780fca70012eb3ccd5698"
server_key_iv: "7947d0cba9fab162779dfa0a3064f9eb"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.236236 8854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:40.265528 8854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:40.270016 8854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:40.270300 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:40.271397 8854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:40.272703 8854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:40.272816 8854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.272962 8854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:40.273043 8854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.305275 8854 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:36291
I20260925 14:58:40.305364 8981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:36291 every 8 connection(s)
I20260925 14:58:40.306625 8854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:40.307961 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8854
I20260925 14:58:40.308123 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:40.308615 5071 external_mini_cluster.cc:1428] Setting key 8764849633ddaad68d2a38c116e77cb2
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:40.327939 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.310258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:43727 (local address 127.4.243.254:37773)
0925 14:58:40.310448 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.310458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.311046 (+ 588us) server_negotiation.cc:407] Connection header received
0925 14:58:40.312906 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.312912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.313002 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.313127 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.314621 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.315585 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.317112 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.317873 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.318026 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.321333 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.321343 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.321350 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.321395 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.324893 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.325595 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.325601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.325605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.325763 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.326155 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.326161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.326168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.326552 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.326737 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.327544 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.327700 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:58:40.328887 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.309802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37773 (local address 127.4.243.194:43727)
0925 14:58:40.310900 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:40.310945 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:40.312620 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:40.313377 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:40.313385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:40.313884 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:40.314430 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.314445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.315740 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.315744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.316969 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.316980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.318099 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.318104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.318321 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.319010 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:40.319030 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:40.321109 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:40.325040 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.325046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.325058 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.325475 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.325873 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.325876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.325878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.326042 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.326719 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:40.326729 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:40.327146 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 14:58:40.327454 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 14:58:40.327825 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":634,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 14:58:40.330403 8982 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37773
I20260925 14:58:40.330724 8982 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:40.331552 8982 heartbeater.cc:511] Master 127.4.243.254:37773 requested a full tablet report, sending...
I20260925 14:58:40.333033 8659 ts_manager.cc:194] Registered new tserver with Master: 734b31813a9f405dbcdf21711206e037 (127.4.243.194:36291)
I20260925 14:58:40.333807 8659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:43727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:40.352059 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37773
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:36763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:40.660774 8987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:40.661056 8987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:40.661206 8987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:40.667918 8987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:40.668058 8987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:40.668215 8987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:40.677419 8987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:36763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:40.678586 8987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:40.679829 8987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:40.692263 8992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.692288 8993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.692368 8995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:40.692724 8987 server_base.cc:1034] running on GCE node
I20260925 14:58:40.693444 8987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:40.694330 8987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:40.695542 8987 hybrid_clock.cc:648] HybridClock initialized: now 1790348320695490 us; error 43 us; skew 500 ppm
I20260925 14:58:40.695844 8987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:40.698743 8987 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:40.699977 8987 webserver.cc:460] Webserver started at http://127.4.243.195:41433/ using document root <none> and password file <none>
I20260925 14:58:40.700629 8987 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:40.700771 8987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:40.701057 8987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:40.703737 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "c78ec2e7e2bf44ac82b247d5e9324149"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-qxh0"
server_key: "78eb310a088418e3871a3149c2a80d91"
server_key_iv: "686bca9583a68a257c96efec4da976a9"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.704439 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "c78ec2e7e2bf44ac82b247d5e9324149"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-qxh0"
server_key: "78eb310a088418e3871a3149c2a80d91"
server_key_iv: "686bca9583a68a257c96efec4da976a9"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.709276 8987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:58:40.712546 9002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.713900 8987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:40.714067 8987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "c78ec2e7e2bf44ac82b247d5e9324149"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-qxh0"
server_key: "78eb310a088418e3871a3149c2a80d91"
server_key_iv: "686bca9583a68a257c96efec4da976a9"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.714267 8987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:40.732043 8987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:40.736153 8987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:40.736424 8987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:40.737309 8987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:40.738515 8987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:40.738656 8987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.738793 8987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:40.738862 8987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.770677 8987 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:41429
I20260925 14:58:40.770777 9114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:41429 every 8 connection(s)
I20260925 14:58:40.771977 8987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:40.778275 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 8987
I20260925 14:58:40.778432 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:40.778903 5071 external_mini_cluster.cc:1428] Setting key 52c11b2022ae32c9ad301b63e88227bb
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:40.791123 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.774500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:60355 (local address 127.4.243.254:37773)
0925 14:58:40.774734 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.774740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.775955 (+ 1215us) server_negotiation.cc:407] Connection header received
0925 14:58:40.777648 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.777652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.777727 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.777813 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.779442 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.780450 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.782380 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.782829 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.782916 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.786004 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.786008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.786011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.786047 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.788570 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.789297 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.789300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.789303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.789404 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.789729 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.789732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.789734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.789989 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.790090 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.790740 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.790883 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:58:40.792053 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.775083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37773 (local address 127.4.243.195:60355)
0925 14:58:40.775804 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:40.775839 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:40.777419 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:40.778010 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:40.778018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:40.778675 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:40.779271 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.779287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.780621 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.780629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.782265 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.782276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.783001 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.783006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.783185 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.783822 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:40.783842 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:40.785832 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:40.788682 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.788688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.788701 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.789135 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.789486 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.789489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.789491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.789644 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.790093 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:40.790098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:40.790391 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 14:58:40.790665 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 14:58:40.791008 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":308,"threads_started":1}
I20260925 14:58:40.793813 9115 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37773
I20260925 14:58:40.794133 9115 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:40.794858 9115 heartbeater.cc:511] Master 127.4.243.254:37773 requested a full tablet report, sending...
I20260925 14:58:40.796164 8658 ts_manager.cc:194] Registered new tserver with Master: c78ec2e7e2bf44ac82b247d5e9324149 (127.4.243.195:41429)
I20260925 14:58:40.796941 8658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:60355
I20260925 14:58:40.806137 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 14:58:40.843374 8848 heartbeater.cc:503] Master 127.4.243.254:37773 was elected leader, sending a full tablet report...
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:40.857028 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.841934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.4.243.193:39753)
0925 14:58:40.842662 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.842668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.842692 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:58:40.842751 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.842756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.842972 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.843140 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.844640 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.845746 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.846996 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.847554 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.847694 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.851406 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.851417 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.851431 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.851468 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.854269 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.854978 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.854985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.854988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.855104 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.855504 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.855509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.855512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.855841 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.856144 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.856432 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.856685 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":332,"threads_started":1}
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:40.874408 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.860971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.4.243.254:37773)
0925 14:58:40.861203 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.861208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.861423 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:58:40.861534 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.861538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.861598 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.861686 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.862927 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.863740 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.864948 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.865366 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.865462 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.869054 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.869060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.869063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.869094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.871801 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.872433 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.872440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.872449 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.872571 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.872986 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.872992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.872995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.873320 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.873560 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.873878 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.874087 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 14:58:40.874922 8658 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58084
I20260925 14:58:40.875873 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8720
I20260925 14:58:40.887156 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.876667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.4.243.193:39753)
0925 14:58:40.876889 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.876895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.877049 (+ 154us) server_negotiation.cc:407] Connection header received
0925 14:58:40.877163 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.877167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.877233 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.877313 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.878388 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.879123 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.880258 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.880641 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.880731 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.882825 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.882829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.882832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.882857 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.884982 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.885552 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.885555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.885557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.885662 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.885945 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.885948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.885950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.886160 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.886328 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.886740 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.886887 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 14:58:40.887755 8843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:40.969123 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8720
I20260925 14:58:40.985865 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8854
Sep 25 14:58:40 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:41.001024 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.986908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.4.243.194:36291)
0925 14:58:40.987455 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.987462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.987486 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:58:40.987543 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.987549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.987780 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.987909 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.989107 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.990128 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.991374 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.991897 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.992011 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.996001 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.996012 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.996027 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.996057 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.998837 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.999400 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.999407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.999409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.999513 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.999818 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.999821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.999823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:41.000112 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:41.000334 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:41.000550 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:58:41.000724 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":270,"threads_started":1}
I20260925 14:58:41.002267 8977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:41.083387 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8854
I20260925 14:58:41.099740 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8987
Sep 25 14:58:41 dist-test-slave-qxh0 krb5kdc[8614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:41.115628 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.100686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.4.243.195:41429)
0925 14:58:41.101299 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.101307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.101335 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:58:41.101390 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.101396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.101607 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.101787 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.103055 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.104064 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.105340 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.105889 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.106032 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.110232 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:41.110242 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:41.110257 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:41.110289 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:41.112983 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:41.113640 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:41.113648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:41.113651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:41.113774 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:41.114175 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:41.114179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:41.114181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:41.114511 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:41.114869 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:41.115128 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 14:58:41.115351 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
I20260925 14:58:41.116904 9110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:41.198467 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8987
I20260925 14:58:41.214474 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 8627
I20260925 14:58:41.215744 8689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:41.315032 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 8627
2026-09-25T14:58:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3732 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:41 dist-test-slave-qxh0 krb5kdc[9134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:41 dist-test-slave-qxh0 krb5kdc[9134](info): set up 2 sockets
Sep 25 14:58:41 dist-test-slave-qxh0 krb5kdc[9134](info): commencing operation
krb5kdc: starting...
W20260925 14:58:42.390933 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:42 dist-test-slave-qxh0 krb5kdc[9134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:42.577809 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43149
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:43149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:42.883502 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:42.883772 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:42.883883 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:42.890851 9147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:42.890945 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:42.891036 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:42.891142 9147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:42.891232 9147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:42.900439 9147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:43149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:42.901666 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:42.902911 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:42.914053 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.914101 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.914290 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:42.915030 9147 server_base.cc:1034] running on GCE node
I20260925 14:58:42.915848 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:42.917196 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:42.918427 9147 hybrid_clock.cc:648] HybridClock initialized: now 1790348322918373 us; error 40 us; skew 500 ppm
I20260925 14:58:42.918839 9147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:42 dist-test-slave-qxh0 krb5kdc[9134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:42.926875 9147 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:42.928014 9147 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:42.928146 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:42.928459 9147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:42.931289 9147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "603c7ede70ef4c55b7ecad69fdbcfde2"
format_stamp: "Formatted at 2026-09-25 14:58:42 on dist-test-slave-qxh0"
server_key: "3d19c33d324d30f2c0a93a01ebed19cd"
server_key_iv: "4158401b1b16ee9e8679492acffe305d"
server_key_version: "encryptionkey@0"
I20260925 14:58:42.932036 9147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "603c7ede70ef4c55b7ecad69fdbcfde2"
format_stamp: "Formatted at 2026-09-25 14:58:42 on dist-test-slave-qxh0"
server_key: "3d19c33d324d30f2c0a93a01ebed19cd"
server_key_iv: "4158401b1b16ee9e8679492acffe305d"
server_key_version: "encryptionkey@0"
I20260925 14:58:42.936964 9147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:58:42.940196 9161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.941445 9147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:42.941622 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "603c7ede70ef4c55b7ecad69fdbcfde2"
format_stamp: "Formatted at 2026-09-25 14:58:42 on dist-test-slave-qxh0"
server_key: "3d19c33d324d30f2c0a93a01ebed19cd"
server_key_iv: "4158401b1b16ee9e8679492acffe305d"
server_key_version: "encryptionkey@0"
I20260925 14:58:42.941874 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:42.960595 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:42.964495 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:42.964761 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:42.982647 9147 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:43149
I20260925 14:58:42.982687 9212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:43149 every 8 connection(s)
I20260925 14:58:42.983529 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:42.987066 9213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:42.993501 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 9147
I20260925 14:58:42.993736 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:42.994365 5071 external_mini_cluster.cc:1428] Setting key 1733e91718671ad8ea83102bc1c733e7
I20260925 14:58:42.997252 9213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2: Bootstrap starting.
I20260925 14:58:43.000681 9213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:43.001916 9213 log.cc:824] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:43.005086 9213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2: No bootstrap required, opened a new log
Sep 25 14:58:43 dist-test-slave-qxh0 krb5kdc[9134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:43.015573 9213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43149 } }
I20260925 14:58:43.016011 9213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:43.016116 9213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603c7ede70ef4c55b7ecad69fdbcfde2, State: Initialized, Role: FOLLOWER
I20260925 14:58:43.016826 9213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43149 } }
I20260925 14:58:43.017088 9213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:43.017230 9213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:43.017411 9213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:43.018427 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.996357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.4.243.254:43149)
0925 14:58:42.997103 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:42.997114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:42.997154 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:58:42.998938 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:42.998960 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:42.999329 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:42.999633 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:43.001525 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.003031 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.004856 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.005578 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.005714 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.010032 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:43.010043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:43.010061 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:43.010104 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:43.013993 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:43.014622 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:43.014637 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:43.014645 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:43.014858 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:43.015201 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:43.015206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:43.015209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:43.015873 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:43.016164 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:43.016646 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 14:58:43.017060 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":357,"threads_started":1}
I20260925 14:58:43.020455 9213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43149 } }
I20260925 14:58:43.020956 9213 leader_election.cc:302] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 603c7ede70ef4c55b7ecad69fdbcfde2; no voters:
I20260925 14:58:43.021567 9213 leader_election.cc:288] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:43.021704 9218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:43.022398 9218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [term 1 LEADER]: Becoming Leader. State: Replica: 603c7ede70ef4c55b7ecad69fdbcfde2, State: Running, Role: LEADER
I20260925 14:58:43.022903 9218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43149 } }
I20260925 14:58:43.023605 9213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:43.025723 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 603c7ede70ef4c55b7ecad69fdbcfde2. Latest consensus state: current_term: 1 leader_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43149 } } }
I20260925 14:58:43.025907 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603c7ede70ef4c55b7ecad69fdbcfde2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43149 } } }
I20260925 14:58:43.026121 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:43.026194 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:43.027771 9223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:43.032637 9223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:43.041976 9223 catalog_manager.cc:1347] Generated new cluster ID: ef422cc8a9ff4cd28706a4ac0f5b3f9a
I20260925 14:58:43.042095 9223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:43.055287 9223 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:43.056555 9223 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:43.066401 9223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2: Generated new TSK 0
I20260925 14:58:43.067207 9223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:43.116318 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43149
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:43.429075 9239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:43.429388 9239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:43.429509 9239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:43.436857 9239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:43.436961 9239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:43.437074 9239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:43.446702 9239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:43.447947 9239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:43.449218 9239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:43.461885 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:43.461925 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:43.461925 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:43.462738 9239 server_base.cc:1034] running on GCE node
I20260925 14:58:43.463315 9239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:43.464264 9239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:43.465493 9239 hybrid_clock.cc:648] HybridClock initialized: now 1790348323465460 us; error 60 us; skew 500 ppm
I20260925 14:58:43.465821 9239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:43 dist-test-slave-qxh0 krb5kdc[9134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348323, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:43.468947 9239 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:43.470073 9239 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:43.470173 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:43.470422 9239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:43.473296 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e"
format_stamp: "Formatted at 2026-09-25 14:58:43 on dist-test-slave-qxh0"
server_key: "203ffd2d0e50b249c4c2b04551263697"
server_key_iv: "4f16abec81409c3f859c083310bd1951"
server_key_version: "encryptionkey@0"
I20260925 14:58:43.474047 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e"
format_stamp: "Formatted at 2026-09-25 14:58:43 on dist-test-slave-qxh0"
server_key: "203ffd2d0e50b249c4c2b04551263697"
server_key_iv: "4f16abec81409c3f859c083310bd1951"
server_key_version: "encryptionkey@0"
I20260925 14:58:43.479182 9239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:58:43.482749 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:43.484050 9239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:58:43.484191 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e"
format_stamp: "Formatted at 2026-09-25 14:58:43 on dist-test-slave-qxh0"
server_key: "203ffd2d0e50b249c4c2b04551263697"
server_key_iv: "4f16abec81409c3f859c083310bd1951"
server_key_version: "encryptionkey@0"
I20260925 14:58:43.484382 9239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:43.503005 9239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:43.507237 9239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:43.507447 9239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:43.508309 9239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:43.509584 9239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:43.509663 9239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:43.509739 9239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:43.509796 9239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:43.542412 9239 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:44363
I20260925 14:58:43.542514 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:44363 every 8 connection(s)
I20260925 14:58:43.543318 9239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:43.543833 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 9239
I20260925 14:58:43.544045 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:43.544958 5071 external_mini_cluster.cc:1428] Setting key 0a15d707247a9863eee89a6f7b0c1cbd
Sep 25 14:58:43 dist-test-slave-qxh0 krb5kdc[9134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348323, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:43.562975 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.546300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:34687 (local address 127.4.243.254:43149)
0925 14:58:43.546778 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:43.546784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:43.547804 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 14:58:43.549550 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:43.549556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:43.549643 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:43.549771 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:43.551291 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.552016 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.553524 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.553910 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.554045 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.557450 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:43.557455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:43.557458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:43.557485 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:43.560015 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:43.560747 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:43.560757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:43.560759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:43.560866 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:43.561277 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:43.561282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:43.561285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:43.561537 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:43.561697 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:43.562352 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 14:58:43.562559 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:43.563946 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.546852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43149 (local address 127.4.243.193:34687)
0925 14:58:43.547661 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:43.547696 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:43.549302 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:43.550010 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:43.550018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:43.550510 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:43.551134 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.551150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.552171 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.552176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.553374 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.553385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.554124 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.554132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.554366 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.555039 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:43.555061 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:43.557228 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:43.560183 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:43.560191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:43.560208 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:43.560621 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:43.560977 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:43.560982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:43.560986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:43.561162 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:43.561676 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:43.561683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:43.562000 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 14:58:43.562256 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 14:58:43.562804 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":358,"threads_started":1}
I20260925 14:58:43.565594 9366 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 14:58:43.565954 9366 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:43.566731 9366 heartbeater.cc:511] Master 127.4.243.254:43149 requested a full tablet report, sending...
I20260925 14:58:43.568778 9178 ts_manager.cc:194] Registered new tserver with Master: 8c60ba6beb0743babf45e0d5c2bb6b1e (127.4.243.193:44363)
I20260925 14:58:43.570086 9178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:34687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:43.589540 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43149
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:43.897019 9372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:43.897280 9372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:43.897372 9372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:43.904001 9372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:43.904103 9372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:43.904275 9372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:43.913372 9372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:43.914809 9372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:43.916028 9372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:43.930873 9378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:43.930944 9380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:43.931420 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:43.931931 9372 server_base.cc:1034] running on GCE node
I20260925 14:58:43.932442 9372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:43.933228 9372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:43.934409 9372 hybrid_clock.cc:648] HybridClock initialized: now 1790348323934353 us; error 51 us; skew 500 ppm
I20260925 14:58:43.934803 9372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:43 dist-test-slave-qxh0 krb5kdc[9134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348323, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:43.937800 9372 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:43.938838 9372 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:43.938917 9372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:43.939240 9372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:43.941847 9372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "19023904893f4feab758f40f47646eac"
format_stamp: "Formatted at 2026-09-25 14:58:43 on dist-test-slave-qxh0"
server_key: "94c623b5e02d05dfa463d91e28a70281"
server_key_iv: "49ea093a45107673e4b0a36c3735d192"
server_key_version: "encryptionkey@0"
I20260925 14:58:43.942636 9372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "19023904893f4feab758f40f47646eac"
format_stamp: "Formatted at 2026-09-25 14:58:43 on dist-test-slave-qxh0"
server_key: "94c623b5e02d05dfa463d91e28a70281"
server_key_iv: "49ea093a45107673e4b0a36c3735d192"
server_key_version: "encryptionkey@0"
I20260925 14:58:43.947449 9372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:58:43.950527 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:43.951740 9372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 14:58:43.951905 9372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "19023904893f4feab758f40f47646eac"
format_stamp: "Formatted at 2026-09-25 14:58:43 on dist-test-slave-qxh0"
server_key: "94c623b5e02d05dfa463d91e28a70281"
server_key_iv: "49ea093a45107673e4b0a36c3735d192"
server_key_version: "encryptionkey@0"
I20260925 14:58:43.952105 9372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:43.976397 9372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:43.980556 9372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:43.980825 9372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:43.981714 9372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:43.983139 9372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:43.983259 9372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:43.983371 9372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:43.983451 9372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:44.016599 9372 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:45465
I20260925 14:58:44.016693 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:45465 every 8 connection(s)
I20260925 14:58:44.017449 9372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:44.026319 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 9372
I20260925 14:58:44.026475 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:44.027033 5071 external_mini_cluster.cc:1428] Setting key beec099fca072ff58e49f334028d28ab
Sep 25 14:58:44 dist-test-slave-qxh0 krb5kdc[9134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348323, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:44.037829 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.019745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:42519 (local address 127.4.243.254:43149)
0925 14:58:44.019944 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.019950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.021057 (+ 1107us) server_negotiation.cc:407] Connection header received
0925 14:58:44.022758 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.022762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.022825 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.022913 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.024384 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.025155 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.027053 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.027506 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.027677 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.032126 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:44.032131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:44.032134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:44.032162 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:44.034813 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.035633 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.035639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.035642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.035772 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.036186 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.036191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.036194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.036452 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:44.036612 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.037257 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.037479 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:58:44.038870 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.020263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43149 (local address 127.4.243.194:42519)
0925 14:58:44.020902 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.020937 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.022497 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.023104 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.023111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.023682 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:44.024209 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.024225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.025281 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.025285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.026925 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.026939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.027772 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.027817 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.028137 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.029184 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:44.029216 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:44.031913 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:44.034952 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.034960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.034976 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.035513 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.035887 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.035892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.035896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.036063 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.036588 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:44.036596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:44.036903 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.037150 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.037650 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
I20260925 14:58:44.040402 9499 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43149
I20260925 14:58:44.040702 9499 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:44.041291 9499 heartbeater.cc:511] Master 127.4.243.254:43149 requested a full tablet report, sending...
I20260925 14:58:44.042727 9178 ts_manager.cc:194] Registered new tserver with Master: 19023904893f4feab758f40f47646eac (127.4.243.194:45465)
I20260925 14:58:44.044129 9178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:42519
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:44.071971 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43149
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:44.384441 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:44.384707 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:44.384807 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:44.392127 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:44.392230 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:44.392338 9504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:44.401703 9504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:44.403146 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:44.404439 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:44.416832 9512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:44.416860 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:44.416983 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:44.417668 9504 server_base.cc:1034] running on GCE node
I20260925 14:58:44.418283 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:44.419216 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:44.420445 9504 hybrid_clock.cc:648] HybridClock initialized: now 1790348324420398 us; error 48 us; skew 500 ppm
I20260925 14:58:44.420784 9504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:44 dist-test-slave-qxh0 krb5kdc[9134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:44.424054 9504 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:44.425237 9504 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:44.425369 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:44.425685 9504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:44.428579 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "327fffcfa8794fd68df2f21481c8d2f6"
format_stamp: "Formatted at 2026-09-25 14:58:44 on dist-test-slave-qxh0"
server_key: "26a1a739c463b41ba05f6f239acc038f"
server_key_iv: "8c11cae28d53108c68dbe5c34c99e4b4"
server_key_version: "encryptionkey@0"
I20260925 14:58:44.429502 9504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "327fffcfa8794fd68df2f21481c8d2f6"
format_stamp: "Formatted at 2026-09-25 14:58:44 on dist-test-slave-qxh0"
server_key: "26a1a739c463b41ba05f6f239acc038f"
server_key_iv: "8c11cae28d53108c68dbe5c34c99e4b4"
server_key_version: "encryptionkey@0"
I20260925 14:58:44.434342 9504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 14:58:44.437891 9518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:44.439517 9504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 14:58:44.439707 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "327fffcfa8794fd68df2f21481c8d2f6"
format_stamp: "Formatted at 2026-09-25 14:58:44 on dist-test-slave-qxh0"
server_key: "26a1a739c463b41ba05f6f239acc038f"
server_key_iv: "8c11cae28d53108c68dbe5c34c99e4b4"
server_key_version: "encryptionkey@0"
I20260925 14:58:44.439917 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:44.462754 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:44.467212 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:44.467483 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:44.468478 9504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:44.469882 9504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:44.470003 9504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:44.470155 9504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:44.470247 9504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:44.501776 9504 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42219
I20260925 14:58:44.501821 9630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42219 every 8 connection(s)
I20260925 14:58:44.502650 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:44.509869 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 9504
I20260925 14:58:44.510037 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:44.510456 5071 external_mini_cluster.cc:1428] Setting key 0c8b8d13ee499e318a754509b0e629a5
Sep 25 14:58:44 dist-test-slave-qxh0 krb5kdc[9134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:44.521155 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.505054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:52933 (local address 127.4.243.254:43149)
0925 14:58:44.505290 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.505294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.506357 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 14:58:44.508133 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.508138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.508222 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.508344 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.509743 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.510581 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.512364 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.512803 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.512896 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.516365 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:44.516369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:44.516372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:44.516402 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:44.518684 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.519351 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.519355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.519357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.519464 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.519811 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.519814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.519816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.520060 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:44.520165 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.520759 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.520907 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 14:58:44.522074 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.505557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43149 (local address 127.4.243.195:52933)
0925 14:58:44.506206 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.506247 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.507869 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.508568 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.508575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.509061 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:44.509589 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.509604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.510753 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.510757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.512237 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.512248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.513013 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.513021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.513352 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.514016 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:44.514036 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:44.516171 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:44.518836 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.518844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.518859 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.519259 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.519550 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.519553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.519555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.519719 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.520172 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:44.520177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:44.520459 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.520684 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.521028 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":255,"threads_started":1}
I20260925 14:58:44.523363 9631 heartbeater.cc:347] Connected to a master server at 127.4.243.254:43149
I20260925 14:58:44.523715 9631 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:44.524402 9631 heartbeater.cc:511] Master 127.4.243.254:43149 requested a full tablet report, sending...
I20260925 14:58:44.525934 9178 ts_manager.cc:194] Registered new tserver with Master: 327fffcfa8794fd68df2f21481c8d2f6 (127.4.243.195:42219)
I20260925 14:58:44.526779 9178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:52933
I20260925 14:58:44.537329 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:44.553705 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.541526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.4.243.254:43149)
0925 14:58:44.541696 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.541700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.542229 (+ 529us) server_negotiation.cc:407] Connection header received
0925 14:58:44.542431 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.542440 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.542495 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.542595 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.544050 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.545140 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.546550 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.547180 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.547306 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.549280 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:44.549284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:44.549287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:44.549315 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:44.551753 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.552265 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.552268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.552270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.552376 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.552671 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.552674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.552675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.552951 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:44.553108 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.553363 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.553496 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:58:44.557600 9178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:44.564141 9178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:44.572654 9366 heartbeater.cc:503] Master 127.4.243.254:43149 was elected leader, sending a full tablet report...
I20260925 14:58:44.598883 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.576549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.4.243.193:44363)
0925 14:58:44.583431 (+ 6882us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.583444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.591342 (+ 7898us) server_negotiation.cc:407] Connection header received
0925 14:58:44.591592 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.591603 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.591883 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.592029 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:44.592687 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.594192 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.596269 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.597654 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.597806 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.598048 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.598312 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.598494 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6651,"thread_start_us":6483,"threads_started":1}
I20260925 14:58:44.598990 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.575954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44363 (local address 127.0.0.1:39802)
0925 14:58:44.591170 (+ 15216us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.591195 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.591362 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.592100 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.592107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.592129 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:44.592503 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.592513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.594323 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.594328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.596120 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.596131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.597867 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.597874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.598048 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.598111 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.598258 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.598446 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15016,"thread_start_us":14818,"threads_started":1}
I20260925 14:58:44.599061 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.575317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42219 (local address 127.0.0.1:59478)
0925 14:58:44.576278 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.576330 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.576522 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.591483 (+ 14961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.591490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.591524 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:44.591957 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.591974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.594081 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.594086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.596430 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.596443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.598149 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.598154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.598332 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.598361 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.598437 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.598545 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":86,"thread_start_us":566,"threads_started":1}
I20260925 14:58:44.600023 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.577155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.4.243.194:45465)
0925 14:58:44.589930 (+ 12775us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.589942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.592269 (+ 2327us) server_negotiation.cc:407] Connection header received
0925 14:58:44.592587 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.592596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.592794 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.592970 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:44.593572 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.595147 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.597246 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.598421 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.598537 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.598786 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.599530 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.599719 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12574,"thread_start_us":12783,"threads_started":1}
I20260925 14:58:44.600502 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.575672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.4.243.195:42219)
0925 14:58:44.590782 (+ 15110us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.590793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.590831 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:58:44.590932 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.590943 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.591195 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.591458 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:44.592209 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.593939 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.596584 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.597933 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.599709 (+ 1776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.599958 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.600007 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.600163 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14885,"thread_start_us":14710,"threads_started":1}
I20260925 14:58:44.600742 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.576637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:45465 (local address 127.0.0.1:38152)
0925 14:58:44.592157 (+ 15520us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.592179 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.592377 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.592959 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.592963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.592987 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:44.593369 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.593380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.595296 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.595301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.597100 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.597110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.599129 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.599140 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.599318 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.599362 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.600296 (+ 934us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.600438 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15390,"thread_start_us":15234,"threads_started":1}
I20260925 14:58:44.603520 9566 tablet_service.cc:1467] Processing CreateTablet for tablet 1450a557b2584af8bca3631a77f318f5 (DEFAULT_TABLE table=test-table [id=9f443837f276499e8b5b0ac5241be566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:44.603538 9301 tablet_service.cc:1467] Processing CreateTablet for tablet 1450a557b2584af8bca3631a77f318f5 (DEFAULT_TABLE table=test-table [id=9f443837f276499e8b5b0ac5241be566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:44.603997 9434 tablet_service.cc:1467] Processing CreateTablet for tablet 1450a557b2584af8bca3631a77f318f5 (DEFAULT_TABLE table=test-table [id=9f443837f276499e8b5b0ac5241be566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:44.604784 9566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1450a557b2584af8bca3631a77f318f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:44.604909 9301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1450a557b2584af8bca3631a77f318f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:44.605139 9434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1450a557b2584af8bca3631a77f318f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:44.615863 9649 tablet_bootstrap.cc:492] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Bootstrap starting.
I20260925 14:58:44.618963 9649 tablet_bootstrap.cc:654] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:44.619118 9650 tablet_bootstrap.cc:492] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Bootstrap starting.
I20260925 14:58:44.620289 9649 log.cc:824] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:44.620723 9651 tablet_bootstrap.cc:492] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Bootstrap starting.
I20260925 14:58:44.622613 9650 tablet_bootstrap.cc:654] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:44.623752 9651 tablet_bootstrap.cc:654] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:44.623780 9650 log.cc:824] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:44.624155 9649 tablet_bootstrap.cc:492] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: No bootstrap required, opened a new log
I20260925 14:58:44.624513 9649 ts_tablet_manager.cc:1397] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260925 14:58:44.624886 9651 log.cc:824] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:44.626717 9650 tablet_bootstrap.cc:492] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: No bootstrap required, opened a new log
I20260925 14:58:44.626958 9650 ts_tablet_manager.cc:1397] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 14:58:44.632447 9651 tablet_bootstrap.cc:492] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: No bootstrap required, opened a new log
I20260925 14:58:44.632681 9651 ts_tablet_manager.cc:1397] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260925 14:58:44.634866 9649 raft_consensus.cc:348] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.635293 9649 raft_consensus.cc:374] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:44.635424 9649 raft_consensus.cc:729] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327fffcfa8794fd68df2f21481c8d2f6, State: Initialized, Role: FOLLOWER
I20260925 14:58:44.636030 9649 consensus_queue.cc:260] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.637115 9631 heartbeater.cc:503] Master 127.4.243.254:43149 was elected leader, sending a full tablet report...
I20260925 14:58:44.637346 9650 raft_consensus.cc:348] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.637750 9650 raft_consensus.cc:374] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:44.637835 9649 ts_tablet_manager.cc:1428] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 14:58:44.637840 9650 raft_consensus.cc:729] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19023904893f4feab758f40f47646eac, State: Initialized, Role: FOLLOWER
I20260925 14:58:44.638473 9650 consensus_queue.cc:260] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.640049 9499 heartbeater.cc:503] Master 127.4.243.254:43149 was elected leader, sending a full tablet report...
I20260925 14:58:44.640726 9650 ts_tablet_manager.cc:1428] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260925 14:58:44.641366 9651 raft_consensus.cc:348] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.641777 9651 raft_consensus.cc:374] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:44.641881 9651 raft_consensus.cc:729] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c60ba6beb0743babf45e0d5c2bb6b1e, State: Initialized, Role: FOLLOWER
I20260925 14:58:44.642412 9651 consensus_queue.cc:260] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.644341 9651 ts_tablet_manager.cc:1428] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 14:58:44.691619 9655 raft_consensus.cc:482] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:44.691920 9655 raft_consensus.cc:504] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.693583 9655 leader_election.cc:288] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19023904893f4feab758f40f47646eac (127.4.243.194:45465), 8c60ba6beb0743babf45e0d5c2bb6b1e (127.4.243.193:44363)
I20260925 14:58:44.700503 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.694034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:36717 (local address 127.4.243.193:44363)
0925 14:58:44.694283 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.694292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.694315 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:58:44.694397 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.694401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.694482 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.694769 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:44.695135 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.696151 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.698662 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.699761 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.699920 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.699995 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.700180 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.700300 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 14:58:44.700548 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.693819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44363 (local address 127.4.243.195:36717)
0925 14:58:44.694007 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.694026 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.694137 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.694755 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.694761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.694796 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:44.694938 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.694970 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.696298 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.696303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.698377 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.698391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.699903 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.699907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.700032 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.700054 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.700126 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.700252 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260925 14:58:44.700981 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.693931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:33899 (local address 127.4.243.194:45465)
0925 14:58:44.694110 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.694115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.694646 (+ 531us) server_negotiation.cc:407] Connection header received
0925 14:58:44.694809 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.694814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.694883 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.694970 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:44.695458 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.696441 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.698942 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.700129 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.700270 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.700328 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.700675 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.700813 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:58:44.701496 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.693829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:45465 (local address 127.4.243.195:33899)
0925 14:58:44.694503 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.694515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.694657 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.695015 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.695021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.695049 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:44.695322 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.695328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.696586 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.696592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.698796 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.698808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.700326 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.700332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.700490 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.700517 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.701188 (+ 671us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.701275 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":173,"thread_start_us":296,"threads_started":1}
I20260925 14:58:44.701447 9321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1450a557b2584af8bca3631a77f318f5" candidate_uuid: "327fffcfa8794fd68df2f21481c8d2f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" is_pre_election: true
I20260925 14:58:44.701933 9321 raft_consensus.cc:2454] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327fffcfa8794fd68df2f21481c8d2f6 in term 0.
I20260925 14:58:44.702508 9454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1450a557b2584af8bca3631a77f318f5" candidate_uuid: "327fffcfa8794fd68df2f21481c8d2f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19023904893f4feab758f40f47646eac" is_pre_election: true
I20260925 14:58:44.702644 9519 leader_election.cc:302] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327fffcfa8794fd68df2f21481c8d2f6, 8c60ba6beb0743babf45e0d5c2bb6b1e; no voters:
I20260925 14:58:44.703006 9655 raft_consensus.cc:2781] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:44.703013 9454 raft_consensus.cc:2454] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327fffcfa8794fd68df2f21481c8d2f6 in term 0.
I20260925 14:58:44.703181 9655 raft_consensus.cc:482] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:44.703315 9655 raft_consensus.cc:3037] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:44.705513 9655 raft_consensus.cc:504] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.706127 9655 leader_election.cc:288] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [CANDIDATE]: Term 1 election: Requested vote from peers 19023904893f4feab758f40f47646eac (127.4.243.194:45465), 8c60ba6beb0743babf45e0d5c2bb6b1e (127.4.243.193:44363)
I20260925 14:58:44.706514 9454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1450a557b2584af8bca3631a77f318f5" candidate_uuid: "327fffcfa8794fd68df2f21481c8d2f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19023904893f4feab758f40f47646eac"
I20260925 14:58:44.706635 9321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1450a557b2584af8bca3631a77f318f5" candidate_uuid: "327fffcfa8794fd68df2f21481c8d2f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e"
I20260925 14:58:44.706744 9454 raft_consensus.cc:3037] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:44.706820 9321 raft_consensus.cc:3037] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:44.709847 9454 raft_consensus.cc:2454] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327fffcfa8794fd68df2f21481c8d2f6 in term 1.
I20260925 14:58:44.709847 9321 raft_consensus.cc:2454] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327fffcfa8794fd68df2f21481c8d2f6 in term 1.
I20260925 14:58:44.710210 9521 leader_election.cc:302] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19023904893f4feab758f40f47646eac, 327fffcfa8794fd68df2f21481c8d2f6; no voters:
I20260925 14:58:44.710489 9655 raft_consensus.cc:2781] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:44.710930 9655 raft_consensus.cc:686] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 1 LEADER]: Becoming Leader. State: Replica: 327fffcfa8794fd68df2f21481c8d2f6, State: Running, Role: LEADER
I20260925 14:58:44.711436 9655 consensus_queue.cc:237] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } }
I20260925 14:58:44.715569 9178 catalog_manager.cc:5495] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 327fffcfa8794fd68df2f21481c8d2f6 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "327fffcfa8794fd68df2f21481c8d2f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327fffcfa8794fd68df2f21481c8d2f6" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:44.740345 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.733930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.4.243.195:42219)
0925 14:58:44.734162 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.734168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.734323 (+ 155us) server_negotiation.cc:407] Connection header received
0925 14:58:44.734563 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.734567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.734706 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.734848 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:44.735196 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.736269 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.738073 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.738637 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.738758 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.739253 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:44.739787 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:44.739959 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.740012 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.740088 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:58:44.748097 9321 raft_consensus.cc:1261] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 1 FOLLOWER]: Refusing update from remote peer 327fffcfa8794fd68df2f21481c8d2f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:44.748227 9454 raft_consensus.cc:1261] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 1 FOLLOWER]: Refusing update from remote peer 327fffcfa8794fd68df2f21481c8d2f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:44.749040 9659 consensus_queue.cc:1035] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c60ba6beb0743babf45e0d5c2bb6b1e" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:44.749374 9655 consensus_queue.cc:1035] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19023904893f4feab758f40f47646eac" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:58:44.754035 9632 tablet.cc:2369] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:44.777262 9177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:44.777567 9177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:44.781289 9177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 603c7ede70ef4c55b7ecad69fdbcfde2: Sending DeleteTablet for 3 replicas of tablet 1450a557b2584af8bca3631a77f318f5
I20260925 14:58:44.782383 9301 tablet_service.cc:1514] Processing DeleteTablet for tablet 1450a557b2584af8bca3631a77f318f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:44 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:39802
I20260925 14:58:44.782406 9434 tablet_service.cc:1514] Processing DeleteTablet for tablet 1450a557b2584af8bca3631a77f318f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:44 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:38152
I20260925 14:58:44.783643 9566 tablet_service.cc:1514] Processing DeleteTablet for tablet 1450a557b2584af8bca3631a77f318f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:44 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:59478
I20260925 14:58:44.784804 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 9239
I20260925 14:58:44.795276 9672 tablet_replica.cc:330] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: stopping tablet replica
I20260925 14:58:44.797240 9673 tablet_replica.cc:330] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: stopping tablet replica
I20260925 14:58:44.797631 9673 raft_consensus.cc:2229] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:44.798128 9673 raft_consensus.cc:2258] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:44.798575 9672 raft_consensus.cc:2229] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:44.799028 9672 raft_consensus.cc:2258] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:44.801065 9672 ts_tablet_manager.cc:1905] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:44.801319 9673 ts_tablet_manager.cc:1905] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:44.808228 9673 ts_tablet_manager.cc:1918] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:44.808394 9673 log.cc:1196] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/1450a557b2584af8bca3631a77f318f5
I20260925 14:58:44.808910 9673 ts_tablet_manager.cc:1939] T 1450a557b2584af8bca3631a77f318f5 P 8c60ba6beb0743babf45e0d5c2bb6b1e: Deleting consensus metadata
I20260925 14:58:44.809701 9672 ts_tablet_manager.cc:1918] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:44.809921 9672 log.cc:1196] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/1450a557b2584af8bca3631a77f318f5
I20260925 14:58:44.810509 9672 ts_tablet_manager.cc:1939] T 1450a557b2584af8bca3631a77f318f5 P 19023904893f4feab758f40f47646eac: Deleting consensus metadata
I20260925 14:58:44.810745 9162 catalog_manager.cc:4841] TS 8c60ba6beb0743babf45e0d5c2bb6b1e (127.4.243.193:44363): tablet 1450a557b2584af8bca3631a77f318f5 (table test-table [id=9f443837f276499e8b5b0ac5241be566]) successfully deleted
I20260925 14:58:44.811496 9674 tablet_replica.cc:330] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: stopping tablet replica
I20260925 14:58:44.811932 9674 raft_consensus.cc:2229] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:44.812582 9674 raft_consensus.cc:2258] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:44.812949 9164 catalog_manager.cc:4841] TS 19023904893f4feab758f40f47646eac (127.4.243.194:45465): tablet 1450a557b2584af8bca3631a77f318f5 (table test-table [id=9f443837f276499e8b5b0ac5241be566]) successfully deleted
I20260925 14:58:44.817107 9674 ts_tablet_manager.cc:1905] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 14:58:44 dist-test-slave-qxh0 krb5kdc[9134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:44.824433 9674 ts_tablet_manager.cc:1918] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:44.824599 9674 log.cc:1196] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/1450a557b2584af8bca3631a77f318f5
I20260925 14:58:44.825167 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.786609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.4.243.193:44363)
0925 14:58:44.786828 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.786837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.809999 (+ 23162us) server_negotiation.cc:407] Connection header received
0925 14:58:44.810698 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.810707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.810805 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.810883 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.812525 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.813526 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.814750 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.815132 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.815276 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.819262 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:44.819273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:44.819287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:44.819322 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:44.822056 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.822813 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.822820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.822822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.822946 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.823349 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.823353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.823356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.823680 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:44.823991 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.824368 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.824893 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:58:44.825150 9674 ts_tablet_manager.cc:1939] T 1450a557b2584af8bca3631a77f318f5 P 327fffcfa8794fd68df2f21481c8d2f6: Deleting consensus metadata
I20260925 14:58:44.825850 9361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:44.827003 9164 catalog_manager.cc:4841] TS 327fffcfa8794fd68df2f21481c8d2f6 (127.4.243.195:42219): tablet 1450a557b2584af8bca3631a77f318f5 (table test-table [id=9f443837f276499e8b5b0ac5241be566]) successfully deleted
I20260925 14:58:44.914532 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 9239
I20260925 14:58:44.932766 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 9372
Sep 25 14:58:44 dist-test-slave-qxh0 krb5kdc[9134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:44.949425 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.933544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.4.243.194:45465)
0925 14:58:44.933710 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.933714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.933748 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:58:44.933978 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.933981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.934044 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.934151 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.935720 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.936492 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.938333 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.938939 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.939032 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.943735 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:44.943748 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:44.943764 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:44.943801 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:44.946850 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.947502 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.947509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.947512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.947637 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.947982 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.947989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.947992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.948333 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:44.948637 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.948891 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.949094 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:58:44.950016 9494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:45.101091 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 9372
I20260925 14:58:45.117871 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 9504
Sep 25 14:58:45 dist-test-slave-qxh0 krb5kdc[9134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:45.136452 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.118878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.4.243.195:42219)
0925 14:58:45.119039 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:45.119043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:45.119087 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:58:45.119141 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:45.119145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:45.119204 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:45.119275 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:45.120925 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.121781 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.123681 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.124170 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.124269 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:45.129418 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:45.129432 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:45.129460 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:45.129524 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:45.133507 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:45.134161 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:45.134173 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:45.134182 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:45.134370 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:45.134780 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:45.134785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:45.134788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:45.135248 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:45.135595 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:45.135805 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 14:58:45.136039 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:58:45.137310 9626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:45.257680 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 9504
I20260925 14:58:45.276666 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 9147
I20260925 14:58:45.277736 9208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:45.382134 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 9147
2026-09-25T14:58:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4066 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T14:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:45Z Disabled control of system clock
I20260925 14:58:45.446815 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37099
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:58:45.750798 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:45.751150 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:45.751295 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:45.758124 9683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:45.758234 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:45.758312 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:45.758406 9683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:45.758478 9683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:45.767123 9683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:37099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:45.768283 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:45.769964 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:45.780998 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:45.780947 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:45.781028 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:45.781729 9683 server_base.cc:1034] running on GCE node
I20260925 14:58:45.782459 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:45.783708 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:45.784904 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790348325784873 us; error 44 us; skew 500 ppm
I20260925 14:58:45.785247 9683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:45.787365 9683 webserver.cc:460] Webserver started at http://127.4.243.254:45621/ using document root <none> and password file <none>
I20260925 14:58:45.787990 9683 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:45.788066 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:45.788364 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:45.791006 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "3d1c00e41ca14922b5ce4cce5a6d183f"
format_stamp: "Formatted at 2026-09-25 14:58:45 on dist-test-slave-qxh0"
server_key: "b9e313be11a09a95f251ce166853ed5e"
server_key_iv: "f0e5722d3f1aae4f206fe15a8ee745ee"
server_key_version: "encryptionkey@0"
I20260925 14:58:45.791719 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "3d1c00e41ca14922b5ce4cce5a6d183f"
format_stamp: "Formatted at 2026-09-25 14:58:45 on dist-test-slave-qxh0"
server_key: "b9e313be11a09a95f251ce166853ed5e"
server_key_iv: "f0e5722d3f1aae4f206fe15a8ee745ee"
server_key_version: "encryptionkey@0"
I20260925 14:58:45.796214 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:58:45.799429 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:45.800556 9683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 14:58:45.800729 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "3d1c00e41ca14922b5ce4cce5a6d183f"
format_stamp: "Formatted at 2026-09-25 14:58:45 on dist-test-slave-qxh0"
server_key: "b9e313be11a09a95f251ce166853ed5e"
server_key_iv: "f0e5722d3f1aae4f206fe15a8ee745ee"
server_key_version: "encryptionkey@0"
I20260925 14:58:45.800925 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:45.806218 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:45.806941 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:45.807219 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:45.824307 9683 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:37099
I20260925 14:58:45.824340 9748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:37099 every 8 connection(s)
I20260925 14:58:45.825551 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:45.828982 9749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:45.832501 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 9683
I20260925 14:58:45.832715 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:45.833470 5071 external_mini_cluster.cc:1428] Setting key 93c939943b8ab0bfd87be43c4279c774
I20260925 14:58:45.840178 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f: Bootstrap starting.
I20260925 14:58:45.842023 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.835508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.4.243.254:37099)
0925 14:58:45.836488 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:45.836498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:45.836546 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:58:45.838785 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:45.838815 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:45.839171 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:45.839457 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:45.840555 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:45.840561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:45.840572 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:45.840586 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:45.840647 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:45.840661 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:45.840825 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:45.840938 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:58:45.841164 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":344,"threads_started":1}
I20260925 14:58:45.843327 9749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:45.844273 9749 log.cc:824] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:45.846743 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f: No bootstrap required, opened a new log
I20260925 14:58:45.853196 9749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37099 } }
I20260925 14:58:45.853528 9749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:45.853626 9749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1c00e41ca14922b5ce4cce5a6d183f, State: Initialized, Role: FOLLOWER
I20260925 14:58:45.854113 9749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37099 } }
I20260925 14:58:45.854334 9749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:45.854463 9749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:45.854622 9749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:45.856608 9749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37099 } }
I20260925 14:58:45.857045 9749 leader_election.cc:302] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d1c00e41ca14922b5ce4cce5a6d183f; no voters:
I20260925 14:58:45.857688 9749 leader_election.cc:288] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:45.857802 9754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:45.858436 9754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [term 1 LEADER]: Becoming Leader. State: Replica: 3d1c00e41ca14922b5ce4cce5a6d183f, State: Running, Role: LEADER
I20260925 14:58:45.859021 9754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37099 } }
I20260925 14:58:45.859771 9749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:45.861797 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37099 } } }
I20260925 14:58:45.861855 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1c00e41ca14922b5ce4cce5a6d183f. Latest consensus state: current_term: 1 leader_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1c00e41ca14922b5ce4cce5a6d183f" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37099 } } }
I20260925 14:58:45.862165 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:45.862236 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:45.863636 9760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:45.868779 9760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:45.876943 9760 catalog_manager.cc:1347] Generated new cluster ID: cf45b598583b4ea294ee037aedb52bbf
I20260925 14:58:45.877068 9760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:45.897027 9760 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:45.897472 9760 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:45.908041 9760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f: Generated new TSK 0
I20260925 14:58:45.908704 9760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:45.912834 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37099
--builtin_ntp_servers=127.4.243.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:58:46.228689 9773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:46.229005 9773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:46.229174 9773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:46.236088 9773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:46.236253 9773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:46.236435 9773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:46.245383 9773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:46.246673 9773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:46.248356 9773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:46.260957 9778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:46.260874 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:46.260926 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:46.262233 9773 server_base.cc:1034] running on GCE node
I20260925 14:58:46.262954 9773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:46.263913 9773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:46.265120 9773 hybrid_clock.cc:648] HybridClock initialized: now 1790348326265051 us; error 65 us; skew 500 ppm
I20260925 14:58:46.265429 9773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:46.267530 9773 webserver.cc:460] Webserver started at http://127.4.243.193:44125/ using document root <none> and password file <none>
I20260925 14:58:46.268205 9773 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:46.268316 9773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:46.268615 9773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:46.271404 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "8300e864ab094cde9fc2f78987ab7adb"
format_stamp: "Formatted at 2026-09-25 14:58:46 on dist-test-slave-qxh0"
server_key: "5d61a5947bfe678b192b07b383836762"
server_key_iv: "40460d48640f9a3ff3c6a655880481d8"
server_key_version: "encryptionkey@0"
I20260925 14:58:46.272143 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "8300e864ab094cde9fc2f78987ab7adb"
format_stamp: "Formatted at 2026-09-25 14:58:46 on dist-test-slave-qxh0"
server_key: "5d61a5947bfe678b192b07b383836762"
server_key_iv: "40460d48640f9a3ff3c6a655880481d8"
server_key_version: "encryptionkey@0"
I20260925 14:58:46.276944 9773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:58:46.280277 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:46.281508 9773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:46.281677 9773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "8300e864ab094cde9fc2f78987ab7adb"
format_stamp: "Formatted at 2026-09-25 14:58:46 on dist-test-slave-qxh0"
server_key: "5d61a5947bfe678b192b07b383836762"
server_key_iv: "40460d48640f9a3ff3c6a655880481d8"
server_key_version: "encryptionkey@0"
I20260925 14:58:46.281876 9773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:46.287249 9773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:46.287964 9773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:46.288213 9773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:46.289044 9773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:46.290339 9773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:46.290422 9773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:46.290583 9773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:46.290740 9773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:46.321202 9773 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:33029
I20260925 14:58:46.321249 9899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:33029 every 8 connection(s)
I20260925 14:58:46.322414 9773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:46.329927 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.324929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:51667 (local address 127.4.243.254:37099)
0925 14:58:46.325142 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:46.325149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:46.326452 (+ 1303us) server_negotiation.cc:407] Connection header received
0925 14:58:46.328088 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:46.328092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:46.328140 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:46.328216 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:46.329328 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:46.329332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:46.329335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:46.329349 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:46.329376 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:46.329386 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:46.329469 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:46.329684 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:58:46.329739 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:58:46.330874 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.325434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37099 (local address 127.4.243.193:51667)
0925 14:58:46.326310 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.326343 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:46.327846 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:46.328401 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:46.328408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:46.328490 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:46.329081 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:46.329083 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:46.329115 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:46.329190 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:46.329490 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:46.329498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:46.329504 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:58:46.329737 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 14:58:46.329904 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":252,"threads_started":1}
I20260925 14:58:46.332255 9900 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37099
I20260925 14:58:46.332506 9900 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:46.332688 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 9773
I20260925 14:58:46.332870 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:46.333091 9900 heartbeater.cc:511] Master 127.4.243.254:37099 requested a full tablet report, sending...
I20260925 14:58:46.333508 5071 external_mini_cluster.cc:1428] Setting key 774b8fbe51d44da133012d99a9a94d48
I20260925 14:58:46.334991 9714 ts_manager.cc:194] Registered new tserver with Master: 8300e864ab094cde9fc2f78987ab7adb (127.4.243.193:33029)
I20260925 14:58:46.336386 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37099
--builtin_ntp_servers=127.4.243.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:58:46.648463 9903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:46.648697 9903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:46.648777 9903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:46.655494 9903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:46.655579 9903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:46.655741 9903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:46.664480 9903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:46.665709 9903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:46.667375 9903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:46.679490 9908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:46.679572 9911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:46.679562 9909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:46.683681 9903 server_base.cc:1034] running on GCE node
I20260925 14:58:46.684160 9903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:46.685070 9903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:46.686309 9903 hybrid_clock.cc:648] HybridClock initialized: now 1790348326686276 us; error 47 us; skew 500 ppm
I20260925 14:58:46.686694 9903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:46.688864 9903 webserver.cc:460] Webserver started at http://127.4.243.194:41565/ using document root <none> and password file <none>
I20260925 14:58:46.689482 9903 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:46.689603 9903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:46.689896 9903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:46.692731 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "031aec524c2b4401bd48787acd5d2ab9"
format_stamp: "Formatted at 2026-09-25 14:58:46 on dist-test-slave-qxh0"
server_key: "89c0628b5be249cf76b31d94a1d630d2"
server_key_iv: "f7933e0fdf6cae3ab939cf1a1c8abab0"
server_key_version: "encryptionkey@0"
I20260925 14:58:46.693440 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "031aec524c2b4401bd48787acd5d2ab9"
format_stamp: "Formatted at 2026-09-25 14:58:46 on dist-test-slave-qxh0"
server_key: "89c0628b5be249cf76b31d94a1d630d2"
server_key_iv: "f7933e0fdf6cae3ab939cf1a1c8abab0"
server_key_version: "encryptionkey@0"
I20260925 14:58:46.698179 9903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 14:58:46.701390 9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:46.702564 9903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:46.702801 9903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "031aec524c2b4401bd48787acd5d2ab9"
format_stamp: "Formatted at 2026-09-25 14:58:46 on dist-test-slave-qxh0"
server_key: "89c0628b5be249cf76b31d94a1d630d2"
server_key_iv: "f7933e0fdf6cae3ab939cf1a1c8abab0"
server_key_version: "encryptionkey@0"
I20260925 14:58:46.703063 9903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:46.708735 9903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:46.709389 9903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:46.709595 9903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:46.710387 9903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:46.711660 9903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:46.711743 9903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:46.711817 9903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:46.711869 9903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:46.745889 9903 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:40411
I20260925 14:58:46.745970 10029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:40411 every 8 connection(s)
I20260925 14:58:46.747131 9903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:46.753683 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 9903
I20260925 14:58:46.753870 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:46.754302 5071 external_mini_cluster.cc:1428] Setting key a3ea48a171c863e55c9937be8bfc1af8
I20260925 14:58:46.755216 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.749476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:40237 (local address 127.4.243.254:37099)
0925 14:58:46.749634 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:46.749638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:46.751068 (+ 1430us) server_negotiation.cc:407] Connection header received
0925 14:58:46.752715 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:46.752720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:46.752793 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:46.752899 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:46.754384 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:46.754391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:46.754395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:46.754414 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:46.754459 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:46.754472 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:46.754597 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:46.754829 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:58:46.754898 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:58:46.756109 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.749968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37099 (local address 127.4.243.194:40237)
0925 14:58:46.750919 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.750953 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:46.752456 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:46.753125 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:46.753133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:46.753222 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:46.754089 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:46.754091 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:46.754127 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:46.754207 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:46.754567 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:46.754577 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:46.754585 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:58:46.754906 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 14:58:46.755144 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
I20260925 14:58:46.757136 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37099
--builtin_ntp_servers=127.4.243.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 14:58:46.757490 10030 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37099
I20260925 14:58:46.757778 10030 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:46.758445 10030 heartbeater.cc:511] Master 127.4.243.254:37099 requested a full tablet report, sending...
I20260925 14:58:46.759908 9714 ts_manager.cc:194] Registered new tserver with Master: 031aec524c2b4401bd48787acd5d2ab9 (127.4.243.194:40411)
W20260925 14:58:47.068893 10033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:47.069140 10033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:47.069295 10033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:47.076018 10033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:47.076112 10033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:47.076301 10033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:47.084728 10033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:47.085930 10033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:47.087677 10033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:47.099670 10038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:47.099828 10039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:47.099699 10041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:47.103864 10033 server_base.cc:1034] running on GCE node
I20260925 14:58:47.104607 10033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:47.105422 10033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:47.106686 10033 hybrid_clock.cc:648] HybridClock initialized: now 1790348327106658 us; error 50 us; skew 500 ppm
I20260925 14:58:47.106980 10033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:47.109004 10033 webserver.cc:460] Webserver started at http://127.4.243.195:43953/ using document root <none> and password file <none>
I20260925 14:58:47.109624 10033 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:47.109746 10033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:47.110033 10033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:47.112630 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "f59233fe66e04e0f87d2b5a50792f0cb"
format_stamp: "Formatted at 2026-09-25 14:58:47 on dist-test-slave-qxh0"
server_key: "2f694889e9baa756bff87fbbab79ff09"
server_key_iv: "cae45e8270988ba40411e098788f3aa7"
server_key_version: "encryptionkey@0"
I20260925 14:58:47.113325 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "f59233fe66e04e0f87d2b5a50792f0cb"
format_stamp: "Formatted at 2026-09-25 14:58:47 on dist-test-slave-qxh0"
server_key: "2f694889e9baa756bff87fbbab79ff09"
server_key_iv: "cae45e8270988ba40411e098788f3aa7"
server_key_version: "encryptionkey@0"
I20260925 14:58:47.117920 10033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 14:58:47.121052 10047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:47.122184 10033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:47.122336 10033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "f59233fe66e04e0f87d2b5a50792f0cb"
format_stamp: "Formatted at 2026-09-25 14:58:47 on dist-test-slave-qxh0"
server_key: "2f694889e9baa756bff87fbbab79ff09"
server_key_iv: "cae45e8270988ba40411e098788f3aa7"
server_key_version: "encryptionkey@0"
I20260925 14:58:47.122543 10033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:47.127743 10033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:47.128429 10033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:47.128684 10033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:47.129473 10033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:47.130697 10033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:47.130806 10033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:47.130940 10033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:47.131021 10033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:47.161969 10033 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:45495
I20260925 14:58:47.162022 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:45495 every 8 connection(s)
I20260925 14:58:47.163236 10033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:47.165156 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10033
I20260925 14:58:47.165350 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:47.165823 5071 external_mini_cluster.cc:1428] Setting key 054362a3c3908d7c95d255918153d523
I20260925 14:58:47.171516 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.165881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:37545 (local address 127.4.243.254:37099)
0925 14:58:47.166113 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.166118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.167886 (+ 1768us) server_negotiation.cc:407] Connection header received
0925 14:58:47.169620 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.169623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.169670 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.169740 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.170951 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.170955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.170958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.170972 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.171000 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.171010 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.171121 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.171269 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.171313 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:58:47.172392 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.166501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37099 (local address 127.4.243.195:37545)
0925 14:58:47.167737 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.167781 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.169382 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.169924 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.169931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.170014 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.170689 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:47.170691 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.170725 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:47.170816 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:47.171110 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:47.171117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:47.171121 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:47.171311 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 14:58:47.171513 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":341,"threads_started":1}
I20260925 14:58:47.173673 10160 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37099
I20260925 14:58:47.173949 10160 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:47.174455 10160 heartbeater.cc:511] Master 127.4.243.254:37099 requested a full tablet report, sending...
I20260925 14:58:47.175657 9714 ts_manager.cc:194] Registered new tserver with Master: f59233fe66e04e0f87d2b5a50792f0cb (127.4.243.195:45495)
I20260925 14:58:47.180680 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:47.187191 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.184739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.4.243.254:37099)
0925 14:58:47.184939 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.184946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.185456 (+ 510us) server_negotiation.cc:407] Connection header received
0925 14:58:47.185602 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.185605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.185648 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.185763 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.186572 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.186576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.186579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.186647 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.186675 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.186687 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.186777 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.186862 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.186903 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:58:47.190456 9714 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:47.196774 9714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:47.212842 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.208211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:45495 (local address 127.0.0.1:56794)
0925 14:58:47.208984 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.209026 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.209153 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.210797 (+ 1644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.210802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.210867 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.211511 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:47.211513 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.211533 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:47.211591 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:47.212104 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:47.212113 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:47.212120 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:58:47.212280 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 14:58:47.212358 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":323,"threads_started":1}
I20260925 14:58:47.213635 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.208369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.4.243.195:45495)
0925 14:58:47.208930 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.208938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.209274 (+ 336us) server_negotiation.cc:407] Connection header received
0925 14:58:47.209343 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.209350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.209616 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.209766 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.211754 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.211764 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.211777 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.211797 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.211866 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.211891 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.212053 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.213178 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.213277 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":410,"threads_started":1}
I20260925 14:58:47.215587 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.210162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.4.243.194:40411)
0925 14:58:47.210713 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.210723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.212068 (+ 1345us) server_negotiation.cc:407] Connection header received
0925 14:58:47.212135 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.212142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.212343 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.212515 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.214820 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.214828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.214839 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.214855 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.214906 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.214929 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.215088 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.215158 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.215243 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":228,"threads_started":1}
I20260925 14:58:47.215660 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.209607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:40411 (local address 127.0.0.1:34996)
0925 14:58:47.211860 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.211876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.211963 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.213422 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.213428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.213451 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.214551 (+ 1100us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:47.214554 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.214565 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:47.214646 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:47.215041 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:47.215045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:47.215050 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:47.215172 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 14:58:47.215220 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2148,"thread_start_us":1659,"threads_started":1}
I20260925 14:58:47.217733 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.209319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57314)
0925 14:58:47.209813 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.209831 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.209909 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.215454 (+ 5545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.215459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.215485 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.216134 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:47.216139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.216154 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:47.216191 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:47.217128 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:47.217133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:47.217138 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:47.217275 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 14:58:47.217324 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":669,"threads_started":1}
I20260925 14:58:47.217274 10095 tablet_service.cc:1467] Processing CreateTablet for tablet a9fba4a787354f3aa6d376cbef950345 (DEFAULT_TABLE table=test-table [id=c76ac80cf5b84f999a87fdde9862fef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:47.217886 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.209835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.4.243.193:33029)
0925 14:58:47.214975 (+ 5140us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.214993 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.215030 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:58:47.215104 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.215113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.215286 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.216073 (+ 787us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.216913 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.216920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.216931 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.216946 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.216991 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.217008 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.217155 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.217417 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.217509 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5004,"thread_start_us":4886,"threads_started":1}
I20260925 14:58:47.218770 10095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fba4a787354f3aa6d376cbef950345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:47.220202 9965 tablet_service.cc:1467] Processing CreateTablet for tablet a9fba4a787354f3aa6d376cbef950345 (DEFAULT_TABLE table=test-table [id=c76ac80cf5b84f999a87fdde9862fef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:47.221526 9965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fba4a787354f3aa6d376cbef950345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:47.221216 9835 tablet_service.cc:1467] Processing CreateTablet for tablet a9fba4a787354f3aa6d376cbef950345 (DEFAULT_TABLE table=test-table [id=c76ac80cf5b84f999a87fdde9862fef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:47.222478 9835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fba4a787354f3aa6d376cbef950345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:47.229879 10178 tablet_bootstrap.cc:492] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Bootstrap starting.
I20260925 14:58:47.233194 10178 tablet_bootstrap.cc:654] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:47.234140 10178 log.cc:824] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:47.235503 10180 tablet_bootstrap.cc:492] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Bootstrap starting.
I20260925 14:58:47.236703 10181 tablet_bootstrap.cc:492] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: Bootstrap starting.
I20260925 14:58:47.237251 10178 tablet_bootstrap.cc:492] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: No bootstrap required, opened a new log
I20260925 14:58:47.237504 10178 ts_tablet_manager.cc:1397] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 14:58:47.239005 10180 tablet_bootstrap.cc:654] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:47.239642 10181 tablet_bootstrap.cc:654] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:47.240290 10180 log.cc:824] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:47.240685 10181 log.cc:824] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:47.243475 10180 tablet_bootstrap.cc:492] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: No bootstrap required, opened a new log
I20260925 14:58:47.243690 10181 tablet_bootstrap.cc:492] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: No bootstrap required, opened a new log
I20260925 14:58:47.243757 10180 ts_tablet_manager.cc:1397] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260925 14:58:47.243923 10181 ts_tablet_manager.cc:1397] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 14:58:47.248473 10178 raft_consensus.cc:348] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.248922 10178 raft_consensus.cc:374] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:47.249068 10178 raft_consensus.cc:729] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59233fe66e04e0f87d2b5a50792f0cb, State: Initialized, Role: FOLLOWER
I20260925 14:58:47.249691 10178 consensus_queue.cc:260] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.250814 10160 heartbeater.cc:503] Master 127.4.243.254:37099 was elected leader, sending a full tablet report...
I20260925 14:58:47.251565 10180 raft_consensus.cc:348] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.251892 10180 raft_consensus.cc:374] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:47.251981 10180 raft_consensus.cc:729] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031aec524c2b4401bd48787acd5d2ab9, State: Initialized, Role: FOLLOWER
I20260925 14:58:47.252605 10180 consensus_queue.cc:260] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.252983 10178 ts_tablet_manager.cc:1428] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260925 14:58:47.253858 10030 heartbeater.cc:503] Master 127.4.243.254:37099 was elected leader, sending a full tablet report...
I20260925 14:58:47.253865 10181 raft_consensus.cc:348] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.254231 10181 raft_consensus.cc:374] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:47.254313 10181 raft_consensus.cc:729] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8300e864ab094cde9fc2f78987ab7adb, State: Initialized, Role: FOLLOWER
I20260925 14:58:47.254412 10180 ts_tablet_manager.cc:1428] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 14:58:47.254980 10181 consensus_queue.cc:260] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.256129 9900 heartbeater.cc:503] Master 127.4.243.254:37099 was elected leader, sending a full tablet report...
I20260925 14:58:47.256688 10181 ts_tablet_manager.cc:1428] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 14:58:47.324573 9901 tablet.cc:2369] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:47.414860 10161 tablet.cc:2369] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:47.498972 10031 tablet.cc:2369] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:47.505856 10186 raft_consensus.cc:482] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:47.506167 10186 raft_consensus.cc:504] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.508023 10186 leader_election.cc:288] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f59233fe66e04e0f87d2b5a50792f0cb (127.4.243.195:45495), 031aec524c2b4401bd48787acd5d2ab9 (127.4.243.194:40411)
I20260925 14:58:47.511433 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.508618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:41777 (local address 127.4.243.195:45495)
0925 14:58:47.508845 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.508853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.509363 (+ 510us) server_negotiation.cc:407] Connection header received
0925 14:58:47.509629 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.509633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.509681 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.509783 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.510828 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.510836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.510842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.510862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.510896 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.510909 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.511012 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.511121 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.511183 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 14:58:47.511489 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.508426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:60861 (local address 127.4.243.194:40411)
0925 14:58:47.508661 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.508670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.509462 (+ 792us) server_negotiation.cc:407] Connection header received
0925 14:58:47.509633 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.509636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.509681 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.509792 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.510892 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.510896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.510899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.510912 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.510940 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.510970 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.511102 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.511192 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.511245 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:58:47.511686 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.508193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:45495 (local address 127.4.243.193:41777)
0925 14:58:47.509235 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.509255 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.509368 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.509820 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.509826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.509847 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.510601 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:47.510625 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.510648 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:47.510690 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:47.511005 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:47.511012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:47.511017 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:47.511147 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 14:58:47.511197 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":675,"threads_started":1}
I20260925 14:58:47.511766 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.508363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:40411 (local address 127.4.243.193:60861)
0925 14:58:47.509299 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.509337 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.509471 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.509809 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.509813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.509836 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.510691 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:47.510693 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.510705 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:47.510759 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:47.511080 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:47.511087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:47.511094 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:58:47.511241 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 14:58:47.511283 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":648,"threads_started":1}
I20260925 14:58:47.512682 9985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fba4a787354f3aa6d376cbef950345" candidate_uuid: "8300e864ab094cde9fc2f78987ab7adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031aec524c2b4401bd48787acd5d2ab9" is_pre_election: true
I20260925 14:58:47.512681 10115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fba4a787354f3aa6d376cbef950345" candidate_uuid: "8300e864ab094cde9fc2f78987ab7adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" is_pre_election: true
I20260925 14:58:47.513123 10115 raft_consensus.cc:2454] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8300e864ab094cde9fc2f78987ab7adb in term 0.
I20260925 14:58:47.513137 9985 raft_consensus.cc:2454] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8300e864ab094cde9fc2f78987ab7adb in term 0.
I20260925 14:58:47.513727 9791 leader_election.cc:302] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 031aec524c2b4401bd48787acd5d2ab9, 8300e864ab094cde9fc2f78987ab7adb, f59233fe66e04e0f87d2b5a50792f0cb; no voters:
I20260925 14:58:47.514120 10186 raft_consensus.cc:2781] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:47.514258 10186 raft_consensus.cc:482] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:47.514371 10186 raft_consensus.cc:3037] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:47.516573 10186 raft_consensus.cc:504] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.517184 10186 leader_election.cc:288] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [CANDIDATE]: Term 1 election: Requested vote from peers f59233fe66e04e0f87d2b5a50792f0cb (127.4.243.195:45495), 031aec524c2b4401bd48787acd5d2ab9 (127.4.243.194:40411)
I20260925 14:58:47.517686 9985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fba4a787354f3aa6d376cbef950345" candidate_uuid: "8300e864ab094cde9fc2f78987ab7adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031aec524c2b4401bd48787acd5d2ab9"
I20260925 14:58:47.517697 10115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fba4a787354f3aa6d376cbef950345" candidate_uuid: "8300e864ab094cde9fc2f78987ab7adb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59233fe66e04e0f87d2b5a50792f0cb"
I20260925 14:58:47.517894 10115 raft_consensus.cc:3037] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:47.517905 9985 raft_consensus.cc:3037] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:47.520179 10115 raft_consensus.cc:2454] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8300e864ab094cde9fc2f78987ab7adb in term 1.
I20260925 14:58:47.520207 9985 raft_consensus.cc:2454] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8300e864ab094cde9fc2f78987ab7adb in term 1.
I20260925 14:58:47.520547 9788 leader_election.cc:302] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8300e864ab094cde9fc2f78987ab7adb, f59233fe66e04e0f87d2b5a50792f0cb; no voters:
I20260925 14:58:47.520824 10186 raft_consensus.cc:2781] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:47.521200 10186 raft_consensus.cc:686] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 1 LEADER]: Becoming Leader. State: Replica: 8300e864ab094cde9fc2f78987ab7adb, State: Running, Role: LEADER
I20260925 14:58:47.521731 10186 consensus_queue.cc:237] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } }
I20260925 14:58:47.525993 9714 catalog_manager.cc:5495] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8300e864ab094cde9fc2f78987ab7adb (127.4.243.193). New cstate: current_term: 1 leader_uuid: "8300e864ab094cde9fc2f78987ab7adb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8300e864ab094cde9fc2f78987ab7adb" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 33029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:47.579504 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.577477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.4.243.193:33029)
0925 14:58:47.577687 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.577693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.577716 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:58:47.577816 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.577819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.577889 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.577997 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.578902 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.578907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.578911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.578933 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.578960 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.578971 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.579101 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.579195 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.579247 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 14:58:47.587468 9985 raft_consensus.cc:1261] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 1 FOLLOWER]: Refusing update from remote peer 8300e864ab094cde9fc2f78987ab7adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:47.587459 10115 raft_consensus.cc:1261] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 1 FOLLOWER]: Refusing update from remote peer 8300e864ab094cde9fc2f78987ab7adb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:47.588430 10189 consensus_queue.cc:1035] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "031aec524c2b4401bd48787acd5d2ab9" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 40411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:47.588799 10186 consensus_queue.cc:1035] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59233fe66e04e0f87d2b5a50792f0cb" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:47.613112 9713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:47.613350 9713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:47.616609 9713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1c00e41ca14922b5ce4cce5a6d183f: Sending DeleteTablet for 3 replicas of tablet a9fba4a787354f3aa6d376cbef950345
I20260925 14:58:47.617610 10095 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fba4a787354f3aa6d376cbef950345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:47 UTC) from {username='slave'} at 127.0.0.1:56794
I20260925 14:58:47.617673 9835 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fba4a787354f3aa6d376cbef950345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:47 UTC) from {username='slave'} at 127.0.0.1:57314
I20260925 14:58:47.618322 9965 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fba4a787354f3aa6d376cbef950345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:47 UTC) from {username='slave'} at 127.0.0.1:34996
I20260925 14:58:47.618671 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 9773
I20260925 14:58:47.621387 10204 tablet_replica.cc:330] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: stopping tablet replica
I20260925 14:58:47.621806 10204 raft_consensus.cc:2229] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:47.622282 10204 raft_consensus.cc:2258] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:47.623649 10202 tablet_replica.cc:330] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb: stopping tablet replica
I20260925 14:58:47.624068 10203 tablet_replica.cc:330] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: stopping tablet replica
I20260925 14:58:47.624432 10203 raft_consensus.cc:2229] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:47.624842 10203 raft_consensus.cc:2258] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:47.625334 10202 raft_consensus.cc:2229] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:47.625646 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.620466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.4.243.193:33029)
0925 14:58:47.620678 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.620687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.623045 (+ 2358us) server_negotiation.cc:407] Connection header received
0925 14:58:47.623218 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.623226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.623303 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.623434 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.624813 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.624819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.624823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.624841 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.624874 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.624887 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.625015 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.625275 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.625336 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:58:47.626080 10202 raft_consensus.cc:2258] T a9fba4a787354f3aa6d376cbef950345 P 8300e864ab094cde9fc2f78987ab7adb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:47.626742 9895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:47.627895 10203 ts_tablet_manager.cc:1905] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:47.628214 10204 ts_tablet_manager.cc:1905] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:47.634689 10203 ts_tablet_manager.cc:1918] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:47.634689 10204 ts_tablet_manager.cc:1918] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:47.634840 10204 log.cc:1196] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/a9fba4a787354f3aa6d376cbef950345
I20260925 14:58:47.634840 10203 log.cc:1196] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/a9fba4a787354f3aa6d376cbef950345
I20260925 14:58:47.635324 10203 ts_tablet_manager.cc:1939] T a9fba4a787354f3aa6d376cbef950345 P f59233fe66e04e0f87d2b5a50792f0cb: Deleting consensus metadata
I20260925 14:58:47.635324 10204 ts_tablet_manager.cc:1939] T a9fba4a787354f3aa6d376cbef950345 P 031aec524c2b4401bd48787acd5d2ab9: Deleting consensus metadata
I20260925 14:58:47.636930 9701 catalog_manager.cc:4841] TS 031aec524c2b4401bd48787acd5d2ab9 (127.4.243.194:40411): tablet a9fba4a787354f3aa6d376cbef950345 (table test-table [id=c76ac80cf5b84f999a87fdde9862fef7]) successfully deleted
I20260925 14:58:47.637012 9698 catalog_manager.cc:4841] TS f59233fe66e04e0f87d2b5a50792f0cb (127.4.243.195:45495): tablet a9fba4a787354f3aa6d376cbef950345 (table test-table [id=c76ac80cf5b84f999a87fdde9862fef7]) successfully deleted
I20260925 14:58:47.720777 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 9773
I20260925 14:58:47.737958 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 9903
W20260925 14:58:47.737967 9698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:33029 (error 108)
I20260925 14:58:47.740020 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.739332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57330)
0925 14:58:47.739793 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.739880 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":247,"threads_started":1}
W20260925 14:58:47.740288 9698 catalog_manager.cc:4584] TS 8300e864ab094cde9fc2f78987ab7adb (127.4.243.193:33029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9fba4a787354f3aa6d376cbef950345: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
I20260925 14:58:47.741070 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.738772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.4.243.194:40411)
0925 14:58:47.738942 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.738950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.739000 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:58:47.739200 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.739204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.739281 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.739393 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.740358 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.740364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.740368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.740388 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.740427 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.740444 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.740589 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.740777 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.740834 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:58:47.741655 10025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:47.793819 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.793384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57336)
0925 14:58:47.793588 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.793682 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 14:58:47.794816 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.794442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57344)
0925 14:58:47.794624 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.794695 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 14:58:47.828415 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 9903
I20260925 14:58:47.845157 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10033
I20260925 14:58:47.847687 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.845928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.4.243.195:45495)
0925 14:58:47.846070 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.846075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.846092 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:58:47.846215 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.846219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.846306 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.846388 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.847215 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:47.847219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:47.847223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:47.847237 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:47.847265 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:47.847276 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:47.847377 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:47.847434 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:58:47.847474 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:47.848240 10155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:47.855913 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.855555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57352)
0925 14:58:47.855721 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.855795 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 14:58:47.856917 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.856580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57360)
0925 14:58:47.856722 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.856797 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:58:47.935868 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10033
I20260925 14:58:47.937495 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.936941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57372)
0925 14:58:47.937196 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.937314 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 14:58:47.938586 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.938207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:33029 (local address 127.0.0.1:57384)
0925 14:58:47.938386 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.938451 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 14:58:47.951972 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 9683
I20260925 14:58:47.953037 9744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:48.053587 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 9683
2026-09-25T14:58:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2670 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 14:58:48.093272 5071 mini_oidc.cc:152] Starting JWKS server
I20260925 14:58:48.095650 5071 webserver.cc:460] Webserver started at https://127.0.0.1:44535/ using document root <none> and password file <none>
I20260925 14:58:48.095880 5071 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:58:48.096437 5071 webserver.cc:460] Webserver started at http://127.0.0.1:43539/ using document root <none> and password file <none>
2026-09-25T14:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:48Z Disabled control of system clock
I20260925 14:58:48.120947 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43925
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:43925
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44535/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/testchainca.cert with env {}
W20260925 14:58:48.424185 10217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:48.424494 10217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:48.424614 10217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:48.431565 10217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:48.431653 10217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:48.431689 10217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:48.431739 10217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:48.431780 10217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:48.431830 10217 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:48.431867 10217 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44535/jwks.json
I20260925 14:58:48.441143 10217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:43925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:43925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44535/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:48.442276 10217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:48.443470 10217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:48.455224 10222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:48.455178 10223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:48.455147 10225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:48.456103 10217 server_base.cc:1034] running on GCE node
I20260925 14:58:48.456828 10217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:48.458171 10217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:48.459405 10217 hybrid_clock.cc:648] HybridClock initialized: now 1790348328459348 us; error 44 us; skew 500 ppm
I20260925 14:58:48.459779 10217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:48.465111 10217 webserver.cc:460] Webserver started at http://127.4.243.254:34653/ using document root <none> and password file <none>
I20260925 14:58:48.465790 10217 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:48.465922 10217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:48.466220 10217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:48.468968 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "f28af0325f724bbfa65eed6388f7f4bb"
format_stamp: "Formatted at 2026-09-25 14:58:48 on dist-test-slave-qxh0"
server_key: "6c2fe5af68a68aa3017d46a3292fa77f"
server_key_iv: "a2a97c1f300c08e26b57df9cf4497a32"
server_key_version: "encryptionkey@0"
I20260925 14:58:48.469730 10217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "f28af0325f724bbfa65eed6388f7f4bb"
format_stamp: "Formatted at 2026-09-25 14:58:48 on dist-test-slave-qxh0"
server_key: "6c2fe5af68a68aa3017d46a3292fa77f"
server_key_iv: "a2a97c1f300c08e26b57df9cf4497a32"
server_key_version: "encryptionkey@0"
I20260925 14:58:48.474366 10217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 14:58:48.477646 10231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:48.479110 10217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:48.479269 10217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "f28af0325f724bbfa65eed6388f7f4bb"
format_stamp: "Formatted at 2026-09-25 14:58:48 on dist-test-slave-qxh0"
server_key: "6c2fe5af68a68aa3017d46a3292fa77f"
server_key_iv: "a2a97c1f300c08e26b57df9cf4497a32"
server_key_version: "encryptionkey@0"
I20260925 14:58:48.479472 10217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:48.510016 10217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:48.510838 10217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:48.511126 10217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:48.528573 10217 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:43925
I20260925 14:58:48.528591 10282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:43925 every 8 connection(s)
I20260925 14:58:48.529846 10217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:48.533504 10283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:48.536550 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10217
I20260925 14:58:48.536782 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:48.537554 5071 external_mini_cluster.cc:1428] Setting key 4605cf85428ca0892b576c8903058d55
I20260925 14:58:48.545341 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb: Bootstrap starting.
I20260925 14:58:48.548420 10283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:48.549463 10283 log.cc:824] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:48.550856 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.539548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.4.243.254:43925)
0925 14:58:48.540437 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:48.540447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:48.540485 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:58:48.542083 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:48.542102 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:48.542533 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:48.542962 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:48.543532 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.544904 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.546261 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.546977 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.547185 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.548279 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:48.548286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:48.548299 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:48.548862 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:48.548960 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:48.548979 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:48.549100 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:48.549301 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 14:58:48.549566 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":314,"threads_started":1}
I20260925 14:58:48.552376 10283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb: No bootstrap required, opened a new log
I20260925 14:58:48.554495 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:48.560518 10283 raft_consensus.cc:348] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28af0325f724bbfa65eed6388f7f4bb" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43925 } }
I20260925 14:58:48.560851 10283 raft_consensus.cc:374] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:48.560969 10283 raft_consensus.cc:729] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28af0325f724bbfa65eed6388f7f4bb, State: Initialized, Role: FOLLOWER
I20260925 14:58:48.561417 10283 consensus_queue.cc:260] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28af0325f724bbfa65eed6388f7f4bb" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43925 } }
I20260925 14:58:48.561611 10283 raft_consensus.cc:388] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:48.561715 10283 raft_consensus.cc:482] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:48.561817 10283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:48.564134 10283 raft_consensus.cc:504] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28af0325f724bbfa65eed6388f7f4bb" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43925 } }
I20260925 14:58:48.564548 10283 leader_election.cc:302] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f28af0325f724bbfa65eed6388f7f4bb; no voters:
I20260925 14:58:48.565027 10283 leader_election.cc:288] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:48.565110 10288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:48.565846 10288 raft_consensus.cc:686] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [term 1 LEADER]: Becoming Leader. State: Replica: f28af0325f724bbfa65eed6388f7f4bb, State: Running, Role: LEADER
I20260925 14:58:48.566282 10288 consensus_queue.cc:237] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28af0325f724bbfa65eed6388f7f4bb" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43925 } }
I20260925 14:58:48.567056 10283 sys_catalog.cc:564] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:48.569010 10290 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f28af0325f724bbfa65eed6388f7f4bb. Latest consensus state: current_term: 1 leader_uuid: "f28af0325f724bbfa65eed6388f7f4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28af0325f724bbfa65eed6388f7f4bb" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43925 } } }
I20260925 14:58:48.569523 10290 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:48.569120 10289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f28af0325f724bbfa65eed6388f7f4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f28af0325f724bbfa65eed6388f7f4bb" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 43925 } } }
I20260925 14:58:48.571242 10289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:48.577845 10304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:48.577956 10304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:48.578104 10297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:48.580086 10297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:48.587821 10297 catalog_manager.cc:1347] Generated new cluster ID: f12d2cda5b60462bbe1dc0f72e58ed54
I20260925 14:58:48.587914 10297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:48.599792 10297 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:48.600888 10297 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:48.611065 10297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f28af0325f724bbfa65eed6388f7f4bb: Generated new TSK 0
I20260925 14:58:48.611739 10297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:48.659118 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.629589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.4.243.254:43925)
0925 14:58:48.629766 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:48.629773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:48.630303 (+ 530us) server_negotiation.cc:407] Connection header received
0925 14:58:48.630755 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:48.630760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:48.630848 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:48.630995 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:48.631543 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.632232 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.634472 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.634958 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.635090 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.635664 (+ 574us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:48.658456 (+ 22792us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:58:48.658834 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35582: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 14:58:48.659250 10286 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35582: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 14:58:48.660877 10312 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:48.629326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:43925 (local address 127.0.0.1:35582)
0925 14:58:48.630203 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:48.630225 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:48.630454 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:48.631024 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:48.631031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:48.631050 (+ 19us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:48.631397 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.631409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.632366 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.632371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.634328 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.634346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.635113 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.635118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.635286 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.635351 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:48.658789 (+ 23438us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:58:48.659120 (+ 331us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.243.254:43925: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":1587,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.243.254:43925: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
E20260925 14:58:48.662081 10313 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 14:58:48.662228 10313 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:58:48.662223 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10217
I20260925 14:58:48.663488 10278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:48.753793 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10217
2026-09-25T14:58:48Z chronyd exiting
I20260925 14:58:48.803057 5071 test_util.cc:175] -----------------------------------------------
I20260925 14:58:48.803169 5071 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284481048-5071-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (713 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 14:58:48.806833 5071 mini_oidc.cc:152] Starting JWKS server
I20260925 14:58:48.808552 5071 webserver.cc:460] Webserver started at https://127.0.0.1:46423/ using document root <none> and password file <none>
I20260925 14:58:48.808781 5071 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:58:48.809360 5071 webserver.cc:460] Webserver started at http://127.0.0.1:36095/ using document root <none> and password file <none>
2026-09-25T14:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:48Z Disabled control of system clock
I20260925 14:58:48.833689 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:40833
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:40833
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46423/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/testchainca.cert with env {}
W20260925 14:58:49.135208 10320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:49.135511 10320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:49.135677 10320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:49.142529 10320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:49.142690 10320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:49.142782 10320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:49.142876 10320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:49.142961 10320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:49.143052 10320 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:49.143110 10320 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46423/jwks.json
I20260925 14:58:49.152437 10320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:40833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46423/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:49.153609 10320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:49.154827 10320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:49.166215 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:49.166179 10328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:49.166246 10326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:49.167129 10320 server_base.cc:1034] running on GCE node
I20260925 14:58:49.167815 10320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:49.169114 10320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:49.170336 10320 hybrid_clock.cc:648] HybridClock initialized: now 1790348329170299 us; error 53 us; skew 500 ppm
I20260925 14:58:49.170710 10320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:49.172714 10320 webserver.cc:460] Webserver started at http://127.4.243.254:36133/ using document root <none> and password file <none>
I20260925 14:58:49.173331 10320 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:49.173454 10320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:49.173763 10320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:49.176366 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "a19050db8d4545ef94eb8ff783a49ede"
format_stamp: "Formatted at 2026-09-25 14:58:49 on dist-test-slave-qxh0"
server_key: "56fb598df71610e9dffbd985353cc7c2"
server_key_iv: "5403ddc76c48c7eeb41ec46e16301ad2"
server_key_version: "encryptionkey@0"
I20260925 14:58:49.177153 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "a19050db8d4545ef94eb8ff783a49ede"
format_stamp: "Formatted at 2026-09-25 14:58:49 on dist-test-slave-qxh0"
server_key: "56fb598df71610e9dffbd985353cc7c2"
server_key_iv: "5403ddc76c48c7eeb41ec46e16301ad2"
server_key_version: "encryptionkey@0"
I20260925 14:58:49.181741 10320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 14:58:49.184994 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:49.186275 10320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:49.186442 10320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "a19050db8d4545ef94eb8ff783a49ede"
format_stamp: "Formatted at 2026-09-25 14:58:49 on dist-test-slave-qxh0"
server_key: "56fb598df71610e9dffbd985353cc7c2"
server_key_iv: "5403ddc76c48c7eeb41ec46e16301ad2"
server_key_version: "encryptionkey@0"
I20260925 14:58:49.186697 10320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:49.204337 10320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:49.205101 10320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:49.205392 10320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:49.222800 10320 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:40833
I20260925 14:58:49.222775 10385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:40833 every 8 connection(s)
I20260925 14:58:49.223964 10320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:49.227350 10386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:49.227872 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10320
I20260925 14:58:49.228111 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:49.228710 5071 external_mini_cluster.cc:1428] Setting key 7cd173a7dd3c3ac3f5d1f3af1f16ede8
I20260925 14:58:49.241775 10386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede: Bootstrap starting.
I20260925 14:58:49.243671 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.230517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.4.243.254:40833)
0925 14:58:49.231694 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:49.231711 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:49.231780 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:58:49.234131 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:49.234156 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:49.234745 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:49.235187 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:49.235663 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.237469 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.238753 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.239791 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.239957 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.241071 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:49.241081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:49.241098 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:49.241805 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:49.241898 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:49.241922 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:49.242046 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:49.242201 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:58:49.242476 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":345,"threads_started":1}
I20260925 14:58:49.245168 10386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:49.246330 10386 log.cc:824] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:49.247308 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:49.248919 10386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede: No bootstrap required, opened a new log
I20260925 14:58:49.256171 10386 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19050db8d4545ef94eb8ff783a49ede" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40833 } }
I20260925 14:58:49.256587 10386 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:49.256711 10386 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19050db8d4545ef94eb8ff783a49ede, State: Initialized, Role: FOLLOWER
I20260925 14:58:49.257272 10386 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19050db8d4545ef94eb8ff783a49ede" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40833 } }
I20260925 14:58:49.257499 10386 raft_consensus.cc:388] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:49.257637 10386 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:49.257809 10386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:49.259876 10386 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19050db8d4545ef94eb8ff783a49ede" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40833 } }
I20260925 14:58:49.260325 10386 leader_election.cc:302] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a19050db8d4545ef94eb8ff783a49ede; no voters:
I20260925 14:58:49.260857 10386 leader_election.cc:288] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:49.261030 10391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:49.261706 10391 raft_consensus.cc:686] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [term 1 LEADER]: Becoming Leader. State: Replica: a19050db8d4545ef94eb8ff783a49ede, State: Running, Role: LEADER
I20260925 14:58:49.262318 10391 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19050db8d4545ef94eb8ff783a49ede" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40833 } }
I20260925 14:58:49.262893 10386 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:49.265367 10392 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a19050db8d4545ef94eb8ff783a49ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19050db8d4545ef94eb8ff783a49ede" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40833 } } }
I20260925 14:58:49.265305 10393 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19050db8d4545ef94eb8ff783a49ede. Latest consensus state: current_term: 1 leader_uuid: "a19050db8d4545ef94eb8ff783a49ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19050db8d4545ef94eb8ff783a49ede" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40833 } } }
I20260925 14:58:49.265782 10392 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:49.265806 10393 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:49.267504 10398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:49.272480 10398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:49.281239 10398 catalog_manager.cc:1347] Generated new cluster ID: f140c6b2673c43639fa062db47d79bb7
I20260925 14:58:49.281358 10398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:49.304601 10398 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:49.306408 10398 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:49.314622 10398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a19050db8d4545ef94eb8ff783a49ede: Generated new TSK 0
I20260925 14:58:49.315521 10398 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:58:49.354115 10416 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 14:58:49.354226 10416 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:58:49.355370 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.335967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.4.243.254:40833)
0925 14:58:49.336154 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:49.336160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:49.336695 (+ 535us) server_negotiation.cc:407] Connection header received
0925 14:58:49.336879 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:49.336885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:49.336970 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:49.337126 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:49.337556 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.338781 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.340115 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.340756 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.340873 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.341296 (+ 423us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:49.354717 (+ 13421us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:58:49.355071 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 14:58:49.355511 10415 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:49.335988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:40833 (local address 127.0.0.1:49190)
0925 14:58:49.336577 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:49.336592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:49.336719 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:49.337137 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:49.337141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:49.337150 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:49.337388 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.337396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.338920 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.338924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.339987 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.339997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.340932 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.340936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.341056 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.341084 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:49.354897 (+ 13813us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:58:49.355114 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.243.254:40833: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":298,"threads_started":1}
W20260925 14:58:49.355500 10389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46423/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 14:58:49.356667 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10320
I20260925 14:58:49.357744 10381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:49.447300 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10320
2026-09-25T14:58:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (694 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 14:58:49.501762 5071 mini_oidc.cc:152] Starting JWKS server
I20260925 14:58:49.503707 5071 webserver.cc:460] Webserver started at https://127.0.0.1:32829/ using document root <none> and password file <none>
I20260925 14:58:49.503931 5071 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:58:49.504557 5071 webserver.cc:460] Webserver started at http://127.0.0.1:43225/ using document root <none> and password file <none>
2026-09-25T14:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:49Z Disabled control of system clock
I20260925 14:58:49.529268 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:41921
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:41921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32829/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:58:49.832228 10423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:49.832485 10423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:49.832614 10423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:49.839389 10423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:49.839478 10423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:49.839514 10423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:49.839567 10423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:49.839612 10423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:49.839663 10423 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:49.839701 10423 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32829/jwks.json
I20260925 14:58:49.848518 10423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:41921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32829/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:49.849635 10423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:49.851305 10423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:49.862246 10429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:49.862332 10431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:49.862296 10428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:49.863245 10423 server_base.cc:1034] running on GCE node
I20260925 14:58:49.863989 10423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:49.865327 10423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:49.866549 10423 hybrid_clock.cc:648] HybridClock initialized: now 1790348329866516 us; error 57 us; skew 500 ppm
I20260925 14:58:49.866937 10423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:49.869068 10423 webserver.cc:460] Webserver started at http://127.4.243.254:41911/ using document root <none> and password file <none>
I20260925 14:58:49.869714 10423 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:49.869843 10423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:49.870154 10423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:49.872874 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "00f0cb95b90d40c7838078ed546a1aee"
format_stamp: "Formatted at 2026-09-25 14:58:49 on dist-test-slave-qxh0"
server_key: "13128c5ad72777b9d8c159f35ab0ca45"
server_key_iv: "d13bf509f0c9d54e8ce0467eeca681f1"
server_key_version: "encryptionkey@0"
I20260925 14:58:49.873646 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "00f0cb95b90d40c7838078ed546a1aee"
format_stamp: "Formatted at 2026-09-25 14:58:49 on dist-test-slave-qxh0"
server_key: "13128c5ad72777b9d8c159f35ab0ca45"
server_key_iv: "d13bf509f0c9d54e8ce0467eeca681f1"
server_key_version: "encryptionkey@0"
I20260925 14:58:49.878384 10423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 14:58:49.881799 10437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:49.883128 10423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:49.883303 10423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "00f0cb95b90d40c7838078ed546a1aee"
format_stamp: "Formatted at 2026-09-25 14:58:49 on dist-test-slave-qxh0"
server_key: "13128c5ad72777b9d8c159f35ab0ca45"
server_key_iv: "d13bf509f0c9d54e8ce0467eeca681f1"
server_key_version: "encryptionkey@0"
I20260925 14:58:49.883498 10423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:49.904886 10423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:49.905665 10423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:49.905958 10423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:49.924635 10423 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:41921
I20260925 14:58:49.924654 10488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:41921 every 8 connection(s)
I20260925 14:58:49.926266 10423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:49.930879 10489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:49.934514 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10423
I20260925 14:58:49.934742 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:49.935528 5071 external_mini_cluster.cc:1428] Setting key 3938a670fd0d5d93f2eb73d9709ae06f
I20260925 14:58:49.943336 10489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee: Bootstrap starting.
I20260925 14:58:49.946724 10489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:49.947841 10489 log.cc:824] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:49.948282 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.937296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.4.243.254:41921)
0925 14:58:49.938236 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:49.938246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:49.938284 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:58:49.939763 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:49.939783 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:49.940215 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:49.940527 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:49.941042 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.942252 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.943940 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.944567 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.944705 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.945874 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:49.945881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:49.945893 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:49.946369 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:49.946449 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:49.946467 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:49.946551 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:49.946843 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 14:58:49.947236 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":351,"threads_started":1}
I20260925 14:58:49.950430 10489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee: No bootstrap required, opened a new log
I20260925 14:58:49.951668 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:49.957952 10489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f0cb95b90d40c7838078ed546a1aee" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41921 } }
I20260925 14:58:49.958287 10489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:49.958389 10489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f0cb95b90d40c7838078ed546a1aee, State: Initialized, Role: FOLLOWER
I20260925 14:58:49.958947 10489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f0cb95b90d40c7838078ed546a1aee" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41921 } }
I20260925 14:58:49.959229 10489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:49.959405 10489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:49.959609 10489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:49.961675 10489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f0cb95b90d40c7838078ed546a1aee" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41921 } }
I20260925 14:58:49.962163 10489 leader_election.cc:302] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f0cb95b90d40c7838078ed546a1aee; no voters:
I20260925 14:58:49.962746 10489 leader_election.cc:288] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:49.962894 10494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:49.963585 10494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [term 1 LEADER]: Becoming Leader. State: Replica: 00f0cb95b90d40c7838078ed546a1aee, State: Running, Role: LEADER
I20260925 14:58:49.964108 10494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f0cb95b90d40c7838078ed546a1aee" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41921 } }
I20260925 14:58:49.964581 10489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:49.966774 10495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00f0cb95b90d40c7838078ed546a1aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f0cb95b90d40c7838078ed546a1aee" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41921 } } }
I20260925 14:58:49.967125 10495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:49.966893 10496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f0cb95b90d40c7838078ed546a1aee. Latest consensus state: current_term: 1 leader_uuid: "00f0cb95b90d40c7838078ed546a1aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f0cb95b90d40c7838078ed546a1aee" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41921 } } }
I20260925 14:58:49.967312 10496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:49.968999 10500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:49.973773 10500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:49.982182 10500 catalog_manager.cc:1347] Generated new cluster ID: 5afdaa0e2bba40f99925fdce40366b32
I20260925 14:58:49.982350 10500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:50.003764 10500 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:50.005103 10500 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:50.016275 10500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00f0cb95b90d40c7838078ed546a1aee: Generated new TSK 0
I20260925 14:58:50.017278 10500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:50.046545 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.038385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.4.243.254:41921)
0925 14:58:50.038585 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:50.038625 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:50.039017 (+ 392us) server_negotiation.cc:407] Connection header received
0925 14:58:50.039195 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:50.039202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:50.039292 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:50.039444 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:50.039852 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.040919 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.042346 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.043070 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.043190 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.043641 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:50.045948 (+ 2307us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:58:50.046250 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 14:58:50.046702 10492 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 14:58:50.048435 10518 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:50.038243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:41921 (local address 127.0.0.1:43590)
0925 14:58:50.038925 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.038940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:50.039051 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:50.039456 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:50.039459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:50.039468 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:50.039709 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.039716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.041143 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.041147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.042215 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.042223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.043227 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.043231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.043352 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.043398 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:50.046152 (+ 2754us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:58:50.048033 (+ 1881us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.243.254:41921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":1590,"thread_start_us":414,"threads_started":1}
I20260925 14:58:50.049983 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10423
I20260925 14:58:50.051146 10484 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 14:58:50.060211 10519 webserver.cc:533] Webserver: sslize failed:
E20260925 14:58:50.060288 10519 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:58:50.143378 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10423
2026-09-25T14:58:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (695 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T14:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:50Z Disabled control of system clock
I20260925 14:58:50.222343 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:40449
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:40449
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/testchainca.cert with env {}
W20260925 14:58:50.530257 10524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:50.530555 10524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:50.530736 10524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:50.537786 10524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:50.537875 10524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:50.537966 10524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:50.538084 10524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:50.538159 10524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:50.538255 10524 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:50.538311 10524 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 14:58:50.547883 10524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:40449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:40449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:50.549170 10524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:50.550395 10524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:50.561563 10530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:50.561612 10529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:50.561635 10532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:50.562709 10524 server_base.cc:1034] running on GCE node
I20260925 14:58:50.563446 10524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:50.564713 10524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:50.565898 10524 hybrid_clock.cc:648] HybridClock initialized: now 1790348330565862 us; error 37 us; skew 500 ppm
I20260925 14:58:50.566228 10524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:58:50.568324 10524 webserver.cc:460] Webserver started at http://127.4.243.254:33837/ using document root <none> and password file <none>
I20260925 14:58:50.568969 10524 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:50.569079 10524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:50.569412 10524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:50.572137 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "14fa1ce371f04fbb981df7b697a7dbbc"
format_stamp: "Formatted at 2026-09-25 14:58:50 on dist-test-slave-qxh0"
server_key: "e50cc852b3f8477329dcbabd397184ca"
server_key_iv: "4c134c1dde78e4c7072000c5624e4312"
server_key_version: "encryptionkey@0"
I20260925 14:58:50.572896 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "14fa1ce371f04fbb981df7b697a7dbbc"
format_stamp: "Formatted at 2026-09-25 14:58:50 on dist-test-slave-qxh0"
server_key: "e50cc852b3f8477329dcbabd397184ca"
server_key_iv: "4c134c1dde78e4c7072000c5624e4312"
server_key_version: "encryptionkey@0"
I20260925 14:58:50.577724 10524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 14:58:50.580915 10538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:50.582154 10524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:50.582333 10524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "14fa1ce371f04fbb981df7b697a7dbbc"
format_stamp: "Formatted at 2026-09-25 14:58:50 on dist-test-slave-qxh0"
server_key: "e50cc852b3f8477329dcbabd397184ca"
server_key_iv: "4c134c1dde78e4c7072000c5624e4312"
server_key_version: "encryptionkey@0"
I20260925 14:58:50.582533 10524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:50.600390 10524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:50.601167 10524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:50.601434 10524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:50.618885 10524 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:40449
I20260925 14:58:50.618916 10589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:40449 every 8 connection(s)
I20260925 14:58:50.620090 10524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:50.623522 10590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:50.628111 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10524
I20260925 14:58:50.628333 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:50.629141 5071 external_mini_cluster.cc:1428] Setting key cf26e27899d26d5903f69097135baee0
I20260925 14:58:50.634454 10590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc: Bootstrap starting.
I20260925 14:58:50.637804 10590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:50.638931 10590 log.cc:824] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:50.641937 10590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc: No bootstrap required, opened a new log
I20260925 14:58:50.642428 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.630969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.4.243.254:40449)
0925 14:58:50.631930 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:50.631949 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:50.631999 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:58:50.633601 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:50.633622 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:50.634052 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:50.634350 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:50.634948 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.636195 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.638069 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.638798 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.638936 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.640218 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:50.640230 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:50.640247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:50.640770 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:50.640837 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:50.640857 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:50.640951 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:50.641202 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 14:58:50.641506 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":337,"threads_started":1}
I20260925 14:58:50.645890 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:50.649595 10590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40449 } }
I20260925 14:58:50.649878 10590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:50.649991 10590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fa1ce371f04fbb981df7b697a7dbbc, State: Initialized, Role: FOLLOWER
I20260925 14:58:50.650486 10590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40449 } }
I20260925 14:58:50.650743 10590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:50.650874 10590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:50.651036 10590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:50.652928 10590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40449 } }
I20260925 14:58:50.653370 10590 leader_election.cc:302] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14fa1ce371f04fbb981df7b697a7dbbc; no voters:
I20260925 14:58:50.653923 10590 leader_election.cc:288] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:50.654114 10595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:50.654830 10595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [term 1 LEADER]: Becoming Leader. State: Replica: 14fa1ce371f04fbb981df7b697a7dbbc, State: Running, Role: LEADER
I20260925 14:58:50.655350 10595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40449 } }
I20260925 14:58:50.655907 10590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:50.658074 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40449 } } }
I20260925 14:58:50.658499 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:50.659058 10597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14fa1ce371f04fbb981df7b697a7dbbc. Latest consensus state: current_term: 1 leader_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fa1ce371f04fbb981df7b697a7dbbc" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40449 } } }
I20260925 14:58:50.659333 10597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:50.660465 10603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:50.665371 10603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:50.673280 10603 catalog_manager.cc:1347] Generated new cluster ID: 27feda866c3643ad8819f699386d14bb
I20260925 14:58:50.673401 10603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:50.684517 10603 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:50.686215 10603 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:50.695318 10603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14fa1ce371f04fbb981df7b697a7dbbc: Generated new TSK 0
I20260925 14:58:50.696125 10603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:50.888314 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.712645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.4.243.254:40449)
0925 14:58:50.712780 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:50.712784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:50.713298 (+ 514us) server_negotiation.cc:407] Connection header received
0925 14:58:50.713593 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:50.713597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:50.713664 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:50.713807 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:50.714271 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.715043 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.716336 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.716757 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.716916 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.717377 (+ 461us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:50.887578 (+170201us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:58:50.888010 (+ 432us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59602: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 14:58:50.888413 10619 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:50.712653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:40449 (local address 127.0.0.1:59602)
0925 14:58:50.713202 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.713217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:50.713387 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:50.713818 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:50.713823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:50.713834 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:50.714119 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.714127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.715180 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.715187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.716188 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.716197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.716942 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.716947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.717107 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.717137 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:50.887876 (+170739us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:58:50.888122 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.243.254:40449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":309,"threads_started":1}
W20260925 14:58:50.888459 10593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59602: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260925 14:58:50.889663 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10524
I20260925 14:58:50.891409 10585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:50.995563 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10524
2026-09-25T14:58:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (833 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (390 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (396 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:51 dist-test-slave-qxh0 krb5kdc[10634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:51 dist-test-slave-qxh0 krb5kdc[10634](info): set up 2 sockets
Sep 25 14:58:51 dist-test-slave-qxh0 krb5kdc[10634](info): commencing operation
krb5kdc: starting...
W20260925 14:58:52.855391 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:52 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:58:53.045483 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36387
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:36387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:53.349483 10647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:53.349800 10647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:53.349952 10647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:53.356956 10647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:53.357091 10647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:53.357198 10647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:53.357297 10647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:53.357370 10647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:53.366526 10647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:36387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:53.367741 10647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:53.368999 10647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:53.379949 10655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:53.380148 10653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:53.380244 10652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:53.381012 10647 server_base.cc:1034] running on GCE node
I20260925 14:58:53.381731 10647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:53.383049 10647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:53.384249 10647 hybrid_clock.cc:648] HybridClock initialized: now 1790348333384197 us; error 42 us; skew 500 ppm
I20260925 14:58:53.384577 10647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:53 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:53.387533 10647 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:58:53.388928 10647 webserver.cc:460] Webserver started at http://127.4.243.254:39331/ using document root <none> and password file <none>
I20260925 14:58:53.389539 10647 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:53.389633 10647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:53.389879 10647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:53.392678 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "c27cd0630fd1468bbbceffe48cc36679"
format_stamp: "Formatted at 2026-09-25 14:58:53 on dist-test-slave-qxh0"
server_key: "cd5a0ed2b5989ab035d37937a31c4ea2"
server_key_iv: "c84c9640c2476e9d41d0cf9574325393"
server_key_version: "encryptionkey@0"
I20260925 14:58:53.393373 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "c27cd0630fd1468bbbceffe48cc36679"
format_stamp: "Formatted at 2026-09-25 14:58:53 on dist-test-slave-qxh0"
server_key: "cd5a0ed2b5989ab035d37937a31c4ea2"
server_key_iv: "c84c9640c2476e9d41d0cf9574325393"
server_key_version: "encryptionkey@0"
I20260925 14:58:53.398186 10647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:58:53.401504 10662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:53.403074 10647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:53.403259 10647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "c27cd0630fd1468bbbceffe48cc36679"
format_stamp: "Formatted at 2026-09-25 14:58:53 on dist-test-slave-qxh0"
server_key: "cd5a0ed2b5989ab035d37937a31c4ea2"
server_key_iv: "c84c9640c2476e9d41d0cf9574325393"
server_key_version: "encryptionkey@0"
I20260925 14:58:53.403455 10647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:53.424278 10647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:53.428802 10647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:53.429122 10647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:53.448727 10647 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:36387
I20260925 14:58:53.448755 10713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:36387 every 8 connection(s)
I20260925 14:58:53.450191 10647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:53.451926 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10647
I20260925 14:58:53.452165 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:53.452929 5071 external_mini_cluster.cc:1428] Setting key e77024f89fb2b09a1ff9531d89366488
I20260925 14:58:53.455070 10714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:53.467681 10714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679: Bootstrap starting.
I20260925 14:58:53.471179 10714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679: Neither blocks nor log segments found. Creating new log.
Sep 25 14:58:53 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:53.472313 10714 log.cc:824] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:53.475219 10714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679: No bootstrap required, opened a new log
I20260925 14:58:53.480186 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:53.455242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.4.243.254:36387)
0925 14:58:53.456080 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:53.456092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:53.456134 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:58:53.457866 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:53.457887 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:53.458298 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:53.458724 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:53.460404 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.461759 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.463528 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.464184 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.464317 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:53.473176 (+ 8859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:53.473184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:53.473199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:53.473234 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:53.476299 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:53.476990 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:53.476997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:53.477004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:53.477164 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:53.477558 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:53.477562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:53.477564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:53.478082 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:53.478395 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:53.478798 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 14:58:53.479219 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":398,"threads_started":1}
I20260925 14:58:53.482878 10714 raft_consensus.cc:348] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27cd0630fd1468bbbceffe48cc36679" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36387 } }
I20260925 14:58:53.483243 10714 raft_consensus.cc:374] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:53.483345 10714 raft_consensus.cc:729] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c27cd0630fd1468bbbceffe48cc36679, State: Initialized, Role: FOLLOWER
I20260925 14:58:53.483865 10714 consensus_queue.cc:260] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27cd0630fd1468bbbceffe48cc36679" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36387 } }
I20260925 14:58:53.484095 10714 raft_consensus.cc:388] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:53.484232 10714 raft_consensus.cc:482] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:53.484398 10714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:53.486449 10714 raft_consensus.cc:504] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27cd0630fd1468bbbceffe48cc36679" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36387 } }
I20260925 14:58:53.486970 10714 leader_election.cc:302] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c27cd0630fd1468bbbceffe48cc36679; no voters:
I20260925 14:58:53.487587 10714 leader_election.cc:288] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:53.487733 10719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:53.488794 10719 raft_consensus.cc:686] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [term 1 LEADER]: Becoming Leader. State: Replica: c27cd0630fd1468bbbceffe48cc36679, State: Running, Role: LEADER
I20260925 14:58:53.489331 10719 consensus_queue.cc:237] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27cd0630fd1468bbbceffe48cc36679" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36387 } }
I20260925 14:58:53.489738 10714 sys_catalog.cc:564] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:53.491969 10721 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27cd0630fd1468bbbceffe48cc36679. Latest consensus state: current_term: 1 leader_uuid: "c27cd0630fd1468bbbceffe48cc36679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27cd0630fd1468bbbceffe48cc36679" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36387 } } }
I20260925 14:58:53.491954 10720 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c27cd0630fd1468bbbceffe48cc36679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27cd0630fd1468bbbceffe48cc36679" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36387 } } }
I20260925 14:58:53.492393 10721 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:53.492393 10720 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:53.493330 10725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:53.498670 10725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:53.507503 10725 catalog_manager.cc:1347] Generated new cluster ID: 5711c8e068fd4eeea6f6a25f006b71e0
I20260925 14:58:53.507603 10725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:53.523784 10725 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:53.524986 10725 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:53.534317 10725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c27cd0630fd1468bbbceffe48cc36679: Generated new TSK 0
I20260925 14:58:53.535166 10725 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:58:53.586189 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36387
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:53.902385 10740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:53.902695 10740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:53.902846 10740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:53.910033 10740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:53.910171 10740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:53.910326 10740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:53.921865 10740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:53.923398 10740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:53.924858 10740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:53.938256 10748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:53.938395 10745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:53.938364 10746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:53.939386 10740 server_base.cc:1034] running on GCE node
I20260925 14:58:53.939975 10740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:53.940779 10740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:53.941975 10740 hybrid_clock.cc:648] HybridClock initialized: now 1790348333941938 us; error 39 us; skew 500 ppm
I20260925 14:58:53.942296 10740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:53 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:53.945528 10740 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:58:53.946866 10740 webserver.cc:460] Webserver started at http://127.4.243.193:38815/ using document root <none> and password file <none>
I20260925 14:58:53.947481 10740 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:53.947573 10740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:53.947799 10740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:53.950457 10740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "ce80bd121652497b823047dffb95e05d"
format_stamp: "Formatted at 2026-09-25 14:58:53 on dist-test-slave-qxh0"
server_key: "22efa579ebb9651868cd1ed2e310de13"
server_key_iv: "5193dddcb09f2e65ce5c16681da7d5d4"
server_key_version: "encryptionkey@0"
I20260925 14:58:53.951212 10740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "ce80bd121652497b823047dffb95e05d"
format_stamp: "Formatted at 2026-09-25 14:58:53 on dist-test-slave-qxh0"
server_key: "22efa579ebb9651868cd1ed2e310de13"
server_key_iv: "5193dddcb09f2e65ce5c16681da7d5d4"
server_key_version: "encryptionkey@0"
I20260925 14:58:53.956146 10740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:58:53.959610 10755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:53.961025 10740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:58:53.961207 10740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "ce80bd121652497b823047dffb95e05d"
format_stamp: "Formatted at 2026-09-25 14:58:53 on dist-test-slave-qxh0"
server_key: "22efa579ebb9651868cd1ed2e310de13"
server_key_iv: "5193dddcb09f2e65ce5c16681da7d5d4"
server_key_version: "encryptionkey@0"
I20260925 14:58:53.961402 10740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:53.975330 10740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:53.979671 10740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:53.979991 10740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:53.980893 10740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:53.982169 10740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:53.982235 10740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:53.982311 10740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:53.982359 10740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:54.014492 10740 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:34963
I20260925 14:58:54.014531 10867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:34963 every 8 connection(s)
I20260925 14:58:54.015792 10740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:54.025311 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10740
I20260925 14:58:54.025537 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:54.026124 5071 external_mini_cluster.cc:1428] Setting key 08c58f53c1934f3242e734f8c93af439
Sep 25 14:58:54 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:54.034834 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.018328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:50963 (local address 127.4.243.254:36387)
0925 14:58:54.018797 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:54.018801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:54.019722 (+ 921us) server_negotiation.cc:407] Connection header received
0925 14:58:54.021491 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:54.021496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:54.021581 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:54.021696 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:54.023164 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.023854 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.025328 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.025845 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.025979 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.029255 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:54.029261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:54.029265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:54.029304 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:54.031767 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:54.032396 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:54.032400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:54.032404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:54.032510 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:54.032920 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:54.032927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:54.032933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:54.033284 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:54.033443 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:54.034222 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 14:58:54.034390 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":268,"threads_started":1}
I20260925 14:58:54.036109 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.018926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36387 (local address 127.4.243.193:50963)
0925 14:58:54.019579 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.019619 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:54.021239 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:54.021946 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:54.021954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:54.022406 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:54.023001 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.023017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.024010 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.024014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.025192 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.025202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.026055 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.026064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.026330 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.026995 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:54.027016 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:54.029072 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:54.031906 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:54.031920 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:54.031935 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:54.032284 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:54.032626 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:54.032632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:54.032635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:54.032811 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:54.033445 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:54.033453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:54.033782 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 14:58:54.034114 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 14:58:54.034712 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260925 14:58:54.037549 10868 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36387
I20260925 14:58:54.037833 10868 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:54.038504 10868 heartbeater.cc:511] Master 127.4.243.254:36387 requested a full tablet report, sending...
I20260925 14:58:54.040359 10679 ts_manager.cc:194] Registered new tserver with Master: ce80bd121652497b823047dffb95e05d (127.4.243.193:34963)
I20260925 14:58:54.041759 10679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:50963
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:58:54.069244 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36387
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:54.373397 10874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:54.373683 10874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:54.373828 10874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:54.380705 10874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:54.380836 10874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:54.380990 10874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:54.390096 10874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:54.391352 10874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:54.392536 10874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:54.405099 10880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:54.405177 10879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:54.405146 10882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:54.405974 10874 server_base.cc:1034] running on GCE node
I20260925 14:58:54.406636 10874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:54.407622 10874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:54.408844 10874 hybrid_clock.cc:648] HybridClock initialized: now 1790348334408786 us; error 51 us; skew 500 ppm
I20260925 14:58:54.409157 10874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:54 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:54.412169 10874 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:58:54.413522 10874 webserver.cc:460] Webserver started at http://127.4.243.194:36773/ using document root <none> and password file <none>
I20260925 14:58:54.414157 10874 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:54.414294 10874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:54.414582 10874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:54.417289 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "e17d16d53af14d149bc83be5fc69709e"
format_stamp: "Formatted at 2026-09-25 14:58:54 on dist-test-slave-qxh0"
server_key: "4708ba91f20ebab5eb479e4527e5f24e"
server_key_iv: "9e20fdab89d050b91101f4b48015c2b9"
server_key_version: "encryptionkey@0"
I20260925 14:58:54.418013 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "e17d16d53af14d149bc83be5fc69709e"
format_stamp: "Formatted at 2026-09-25 14:58:54 on dist-test-slave-qxh0"
server_key: "4708ba91f20ebab5eb479e4527e5f24e"
server_key_iv: "9e20fdab89d050b91101f4b48015c2b9"
server_key_version: "encryptionkey@0"
I20260925 14:58:54.422706 10874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260925 14:58:54.425776 10889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:54.427080 10874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:54.427243 10874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "e17d16d53af14d149bc83be5fc69709e"
format_stamp: "Formatted at 2026-09-25 14:58:54 on dist-test-slave-qxh0"
server_key: "4708ba91f20ebab5eb479e4527e5f24e"
server_key_iv: "9e20fdab89d050b91101f4b48015c2b9"
server_key_version: "encryptionkey@0"
I20260925 14:58:54.427425 10874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:54.444625 10874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:54.449064 10874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:54.449334 10874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:54.450361 10874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:54.451848 10874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:54.451942 10874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:54.452075 10874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:54.452131 10874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:54.488773 10874 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:45417
I20260925 14:58:54.488818 11001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:45417 every 8 connection(s)
I20260925 14:58:54.489954 10874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:54.495078 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 10874
I20260925 14:58:54.495260 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:54.495739 5071 external_mini_cluster.cc:1428] Setting key 6d2290bbd824909fc16db46f0dcfd864
Sep 25 14:58:54 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:54.511053 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.492413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:38797 (local address 127.4.243.254:36387)
0925 14:58:54.492637 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:54.492643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:54.493734 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 14:58:54.495632 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:54.495637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:54.495733 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:54.495835 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:54.497659 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.498732 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.500260 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.500990 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.501122 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.504551 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:54.504559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:54.504566 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:54.504606 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:54.508109 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:54.508759 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:54.508766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:54.508772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:54.508928 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:54.509280 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:54.509287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:54.509293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:54.509645 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:54.509797 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:54.510355 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 14:58:54.510599 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:58:54.511641 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.492908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36387 (local address 127.4.243.194:38797)
0925 14:58:54.493588 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.493623 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:54.495368 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:54.496096 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:54.496103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:54.496735 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:54.497475 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.497498 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.498878 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.498881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.500115 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.500126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.501183 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.501188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.501369 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.502026 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:54.502050 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:54.504359 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:54.508232 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:54.508238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:54.508249 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:54.508661 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:54.509026 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:54.509030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:54.509032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:54.509181 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:54.509759 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:54.509764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:54.510027 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 14:58:54.510247 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 14:58:54.510586 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260925 14:58:54.513034 11002 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36387
I20260925 14:58:54.513342 11002 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:54.514284 11002 heartbeater.cc:511] Master 127.4.243.254:36387 requested a full tablet report, sending...
I20260925 14:58:54.515766 10679 ts_manager.cc:194] Registered new tserver with Master: e17d16d53af14d149bc83be5fc69709e (127.4.243.194:45417)
I20260925 14:58:54.516508 10679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:38797
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:58:54.539953 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36387
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:54.845499 11007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:54.845752 11007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:54.845846 11007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:54.852622 11007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:54.852712 11007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:54.852815 11007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:54.861699 11007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:54.862826 11007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:54.863967 11007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:54.876368 11015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:54.876482 11012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:54.876485 11013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:54.877331 11007 server_base.cc:1034] running on GCE node
I20260925 14:58:54.877872 11007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:54.878775 11007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:54.879993 11007 hybrid_clock.cc:648] HybridClock initialized: now 1790348334879957 us; error 46 us; skew 500 ppm
I20260925 14:58:54.880302 11007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:54 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:54.883056 11007 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:58:54.884246 11007 webserver.cc:460] Webserver started at http://127.4.243.195:39779/ using document root <none> and password file <none>
I20260925 14:58:54.884776 11007 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:54.884866 11007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:54.885159 11007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:54.887928 11007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "4bebc9f10ff64d58945729cdf7229599"
format_stamp: "Formatted at 2026-09-25 14:58:54 on dist-test-slave-qxh0"
server_key: "2d0a04b195f6e588e773446de50cb7a4"
server_key_iv: "53344abdfd347e1c8ec8512a5d0c0a72"
server_key_version: "encryptionkey@0"
I20260925 14:58:54.888622 11007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "4bebc9f10ff64d58945729cdf7229599"
format_stamp: "Formatted at 2026-09-25 14:58:54 on dist-test-slave-qxh0"
server_key: "2d0a04b195f6e588e773446de50cb7a4"
server_key_iv: "53344abdfd347e1c8ec8512a5d0c0a72"
server_key_version: "encryptionkey@0"
I20260925 14:58:54.893358 11007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:58:54.896467 11022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:54.897691 11007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 14:58:54.897811 11007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "4bebc9f10ff64d58945729cdf7229599"
format_stamp: "Formatted at 2026-09-25 14:58:54 on dist-test-slave-qxh0"
server_key: "2d0a04b195f6e588e773446de50cb7a4"
server_key_iv: "53344abdfd347e1c8ec8512a5d0c0a72"
server_key_version: "encryptionkey@0"
I20260925 14:58:54.897974 11007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:54.928514 11007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:54.934140 11007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:54.934422 11007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:54.935568 11007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:54.936941 11007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:54.937017 11007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:54.937175 11007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:54.937232 11007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:54.970322 11007 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42839
I20260925 14:58:54.970366 11134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42839 every 8 connection(s)
I20260925 14:58:54.971576 11007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:54.975808 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11007
I20260925 14:58:54.976011 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:54.976580 5071 external_mini_cluster.cc:1428] Setting key 07202e9bbfdccfa2cd596e47cf269d8e
Sep 25 14:58:54 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:58:54.991364 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.974102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:49183 (local address 127.4.243.254:36387)
0925 14:58:54.974266 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:54.974270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:54.975426 (+ 1156us) server_negotiation.cc:407] Connection header received
0925 14:58:54.977793 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:54.977798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:54.977876 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:54.977990 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:54.979774 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.980531 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.981931 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.982339 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.982427 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.985573 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:54.985578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:54.985582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:54.985611 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:54.988190 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:54.988886 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:54.988889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:54.988892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:54.988996 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:54.989367 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:54.989374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:54.989381 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:54.989643 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:54.989749 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:54.990745 (+ 996us) server_negotiation.cc:299] Negotiation successful
0925 14:58:54.990966 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:58:54.992353 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.974487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36387 (local address 127.4.243.195:49183)
0925 14:58:54.975241 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.975283 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:54.977528 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:54.978183 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:54.978192 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:54.978929 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:54.979624 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.979643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.980655 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.980658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.981816 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.981827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.982509 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.982513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.982748 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.983399 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:54.983420 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:54.985387 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:54.988345 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:54.988351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:54.988362 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:54.988780 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:54.989085 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:54.989088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:54.989090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:54.989248 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:54.989816 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:54.989823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:54.990206 (+ 383us) client_negotiation.cc:769] Sending connection context
0925 14:58:54.990588 (+ 382us) client_negotiation.cc:240] Negotiation successful
0925 14:58:54.991109 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":292,"threads_started":1}
I20260925 14:58:54.993593 11135 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36387
I20260925 14:58:54.993846 11135 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:54.994474 11135 heartbeater.cc:511] Master 127.4.243.254:36387 requested a full tablet report, sending...
I20260925 14:58:54.995790 10679 ts_manager.cc:194] Registered new tserver with Master: 4bebc9f10ff64d58945729cdf7229599 (127.4.243.195:42839)
I20260925 14:58:54.996534 10679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:49183
I20260925 14:58:55.003863 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/test-admin.keytab.
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:55.029513 5071 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 14:58:55.034287 11142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:55.032006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.4.243.193:34963)
0925 14:58:55.032627 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.032634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.032661 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:58:55.032771 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.032776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.032982 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.033105 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.033913 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44652: BlockingRecv error: recv got EOF from 127.0.0.1:44652 (error 108)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":269,"threads_started":1}
W20260925 14:58:55.034337 5071 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:55.036504 5071 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:55.037837 5071 init.cc:295] Successfully reacquired a new kerberos TGT
I20260925 14:58:55.044394 10868 heartbeater.cc:503] Master 127.4.243.254:36387 was elected leader, sending a full tablet report...
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:55.052349 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.039377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.4.243.193:34963)
0925 14:58:55.039558 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.039564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.039719 (+ 155us) server_negotiation.cc:407] Connection header received
0925 14:58:55.039844 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.039847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.039897 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.039980 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.040894 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.041823 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.043049 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.043582 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.043681 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.047605 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:55.047615 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:55.047628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:55.047663 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:55.050081 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.050675 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.050683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.050687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.050806 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.051115 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.051118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.051120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.051419 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:55.051611 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:55.051851 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 14:58:55.052114 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 14:58:55.057693 11142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:55.056072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.4.243.193:34963)
0925 14:58:55.056282 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.056287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.056517 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:58:55.056684 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.056695 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.056756 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.056853 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.057464 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44670: BlockingRecv error: recv got EOF from 127.0.0.1:44670 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 14:58:55.058038 5071 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc)
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:55.059468 5071 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:55.060696 5071 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:55.073447 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.061856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.4.243.193:34963)
0925 14:58:55.061992 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.061996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.062254 (+ 258us) server_negotiation.cc:407] Connection header received
0925 14:58:55.062355 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.062359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.062404 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.062477 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.063398 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.064130 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.065310 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.065709 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.065809 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.069026 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:55.069030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:55.069033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:55.069069 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:55.071597 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.072109 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.072113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.072115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.072215 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.072502 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.072505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.072506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.072733 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:55.072913 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:55.073125 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 14:58:55.073253 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 14:58:55.077253 11142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:55.075650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.4.243.193:34963)
0925 14:58:55.075829 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.075835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.076034 (+ 199us) server_negotiation.cc:407] Connection header received
0925 14:58:55.076211 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.076216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.076287 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.076381 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.077013 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44674: BlockingRecv error: recv got EOF from 127.0.0.1:44674 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 14:58:55.077448 5071 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284481048-5071-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:55.078866 5071 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:55.080024 5071 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:58:55.093060 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.081210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.4.243.193:34963)
0925 14:58:55.081388 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.081392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.081488 (+ 96us) server_negotiation.cc:407] Connection header received
0925 14:58:55.081587 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.081590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.081635 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.081704 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.082515 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.083257 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.084357 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.084733 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.084822 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.088192 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:55.088197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:55.088200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:55.088226 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:55.090664 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.091376 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.091389 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.091392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.091527 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.091869 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.091874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.091876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.092142 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:55.092363 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:55.092577 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 14:58:55.092773 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:58:55.094506 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10740
I20260925 14:58:55.105428 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.095158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.4.243.193:34963)
0925 14:58:55.095277 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.095280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.095696 (+ 416us) server_negotiation.cc:407] Connection header received
0925 14:58:55.095821 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.095824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.095869 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.095945 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.096707 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.097332 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.098441 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.098873 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.098968 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.100814 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:55.100822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:55.100826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:55.100865 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:55.103232 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.103838 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.103842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.103844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.103972 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.104317 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.104320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.104323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.104574 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:55.104751 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:55.105040 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 14:58:55.105182 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:55.105933 10863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:55.187062 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10740
I20260925 14:58:55.203603 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10874
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:58:55.218183 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.204533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.4.243.194:45417)
0925 14:58:55.205093 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.205100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.205124 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:58:55.205186 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.205193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.205405 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.205556 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.206522 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.207708 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.208942 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.209465 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.209564 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.213108 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:55.213117 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:55.213130 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:55.213157 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:55.215792 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.216358 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.216364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.216366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.216492 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.216788 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.216792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.216794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.217097 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:55.217307 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:55.217550 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:58:55.217873 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":285,"threads_started":1}
I20260925 14:58:55.219501 10997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:55.302099 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10874
I20260925 14:58:55.317651 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11007
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:58:55.332593 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.318674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.4.243.195:42839)
0925 14:58:55.319293 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.319300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.319328 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:58:55.319384 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.319390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.319615 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.319732 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.320648 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.321671 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.322988 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.323523 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.323620 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.327474 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:55.327485 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:55.327498 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:55.327527 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:55.330326 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.330994 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.331001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.331003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.331119 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:55.331410 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:55.331413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:55.331414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:55.331707 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:55.331923 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:55.332149 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 14:58:55.332326 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":334,"threads_started":1}
I20260925 14:58:55.333802 11130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:55.416760 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11007
I20260925 14:58:55.432526 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 10647
I20260925 14:58:55.433683 10709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:58:55.533274 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 10647
2026-09-25T14:58:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3753 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[11162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[11162](info): set up 2 sockets
Sep 25 14:58:55 dist-test-slave-qxh0 krb5kdc[11162](info): commencing operation
krb5kdc: starting...
W20260925 14:58:56.619306 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:58:56 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab.
Entry for principal oryx/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab.
I20260925 14:58:56.812076 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38093
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab
--principal=oryx/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:38093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:57.120155 11175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.120429 11175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.120620 11175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.127514 11175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:57.127600 11175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.127701 11175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.127782 11175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:57.127877 11175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:57.137019 11175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:38093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab
--principal=oryx/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:57.138285 11175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.139611 11175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.150628 11181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.150636 11180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.150640 11183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.151387 11175 server_base.cc:1034] running on GCE node
I20260925 14:58:57.152093 11175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.153335 11175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.154557 11175 hybrid_clock.cc:648] HybridClock initialized: now 1790348337154519 us; error 46 us; skew 500 ppm
I20260925 14:58:57.154897 11175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:57 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348337, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:57.157833 11175 init.cc:377] Logged in from keytab as oryx/127.4.243.254@KRBTEST.COM (short username oryx)
I20260925 14:58:57.159205 11175 webserver.cc:460] Webserver started at http://127.4.243.254:45449/ using document root <none> and password file <none>
I20260925 14:58:57.159822 11175 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.159970 11175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.160249 11175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.163077 11175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "6b4bf6ce139749de929c37851c302282"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-qxh0"
server_key: "12320d25fc698cf840faa935fac367a0"
server_key_iv: "1d8a35c05a7162fd0a0ccf08a83e61b7"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.163789 11175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "6b4bf6ce139749de929c37851c302282"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-qxh0"
server_key: "12320d25fc698cf840faa935fac367a0"
server_key_iv: "1d8a35c05a7162fd0a0ccf08a83e61b7"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.168622 11175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:58:57.171957 11190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.173233 11175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:58:57.173404 11175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "6b4bf6ce139749de929c37851c302282"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-qxh0"
server_key: "12320d25fc698cf840faa935fac367a0"
server_key_iv: "1d8a35c05a7162fd0a0ccf08a83e61b7"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.173592 11175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.199793 11175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.203573 11175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.203847 11175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.221778 11175 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:38093
I20260925 14:58:57.221809 11241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:38093 every 8 connection(s)
I20260925 14:58:57.223073 11175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:57.226529 11242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:57.227931 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11175
I20260925 14:58:57.228152 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:58:57.228785 5071 external_mini_cluster.cc:1428] Setting key 3818270fd643a6d26ad0831fd0e94d8a
I20260925 14:58:57.237677 11242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: Bootstrap starting.
I20260925 14:58:57.241322 11242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:57.242463 11242 log.cc:824] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:58:57 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:58:57.245172 11242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: No bootstrap required, opened a new log
I20260925 14:58:57.250473 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.230733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.4.243.254:38093)
0925 14:58:57.231482 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.231496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.231532 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:58:57.233043 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.233061 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.233405 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.233677 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.235376 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.236596 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.238261 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.238944 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.239089 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.243662 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.243670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.243683 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:57.243723 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.246682 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:57.247351 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:57.247357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:57.247364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:57.247519 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:57.247908 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:57.247912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:57.247914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:57.248402 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.248703 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.249157 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.249539 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":353,"threads_started":1}
I20260925 14:58:57.253263 11242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } }
I20260925 14:58:57.253626 11242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:57.253727 11242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4bf6ce139749de929c37851c302282, State: Initialized, Role: FOLLOWER
I20260925 14:58:57.254228 11242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } }
I20260925 14:58:57.254439 11242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:57.254575 11242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:57.254757 11242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:57.256816 11242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } }
I20260925 14:58:57.257368 11242 leader_election.cc:302] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4bf6ce139749de929c37851c302282; no voters:
I20260925 14:58:57.257956 11242 leader_election.cc:288] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:57.258097 11247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:57.258734 11247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 1 LEADER]: Becoming Leader. State: Replica: 6b4bf6ce139749de929c37851c302282, State: Running, Role: LEADER
I20260925 14:58:57.259238 11247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } }
I20260925 14:58:57.259898 11242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:57.261623 11248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } }
I20260925 14:58:57.261929 11248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:57.262614 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4bf6ce139749de929c37851c302282. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } }
I20260925 14:58:57.262895 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:57.262737 11253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:57.267781 11253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:57.276558 11253 catalog_manager.cc:1347] Generated new cluster ID: f8d2a6c02ade41339d86bfab51123204
I20260925 14:58:57.276665 11253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:57.286628 11253 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:57.288360 11253 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:57.299554 11253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: Generated new TSK 0
I20260925 14:58:57.300565 11253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab.
Entry for principal oryx/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab.
I20260925 14:58:57.350579 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38093
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab
--principal=oryx/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:57.668607 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.668910 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.669080 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.676337 11268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.676472 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.676636 11268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:58:57.685989 11268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab
--principal=oryx/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:57.687266 11268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.688519 11268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.700774 11276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.700834 11273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.700800 11274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.701519 11268 server_base.cc:1034] running on GCE node
I20260925 14:58:57.702049 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.702930 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.704151 11268 hybrid_clock.cc:648] HybridClock initialized: now 1790348337704111 us; error 53 us; skew 500 ppm
I20260925 14:58:57.704437 11268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:57 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348337, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:57.707414 11268 init.cc:377] Logged in from keytab as oryx/127.4.243.193@KRBTEST.COM (short username oryx)
I20260925 14:58:57.708788 11268 webserver.cc:460] Webserver started at http://127.4.243.193:37515/ using document root <none> and password file <none>
I20260925 14:58:57.709444 11268 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.709590 11268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.709879 11268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.712735 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "b08a3dae84034ac2aba9be8b32c40894"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-qxh0"
server_key: "bd4c7441c66967a35ed9da434b921cd0"
server_key_iv: "7eea4eaf8390caf912980f31296a1bb0"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.713451 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "b08a3dae84034ac2aba9be8b32c40894"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-qxh0"
server_key: "bd4c7441c66967a35ed9da434b921cd0"
server_key_iv: "7eea4eaf8390caf912980f31296a1bb0"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.718374 11268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 14:58:57.721627 11283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.722918 11268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:58:57.723079 11268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "b08a3dae84034ac2aba9be8b32c40894"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-qxh0"
server_key: "bd4c7441c66967a35ed9da434b921cd0"
server_key_iv: "7eea4eaf8390caf912980f31296a1bb0"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.723264 11268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.735452 11268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.739116 11268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.739389 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.740243 11268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:57.741513 11268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:57.741607 11268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.741750 11268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:57.741827 11268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.774267 11268 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:39633
I20260925 14:58:57.774351 11395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:39633 every 8 connection(s)
I20260925 14:58:57.775627 11268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:57.781729 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11268
I20260925 14:58:57.781955 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:57.782570 5071 external_mini_cluster.cc:1428] Setting key 97665e6bec434d8974f3f06961b836fa
Sep 25 14:58:57 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348337, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:58:57.796981 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.778222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:47871 (local address 127.4.243.254:38093)
0925 14:58:57.778672 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.778677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.779649 (+ 972us) server_negotiation.cc:407] Connection header received
0925 14:58:57.781351 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.781358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.781445 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.781588 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.783189 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.784046 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.786572 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.787242 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.787394 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.791443 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.791447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.791450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:57.791482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.793987 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:57.794738 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:57.794742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:57.794745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:57.794854 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:57.795274 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:57.795279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:57.795282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:57.795558 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.795709 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.796404 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.796626 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":212,"threads_started":1}
I20260925 14:58:57.797959 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.778847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.193:47871)
0925 14:58:57.779498 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.779538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:57.781091 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:57.781762 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:57.781770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:57.782338 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:57.783012 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.783029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.784235 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.784240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.786381 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.786395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.787495 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.787504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.787819 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.788554 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:57.788587 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:57.791218 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:57.794136 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:57.794145 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:57.794163 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:57.794582 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:57.794964 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:57.794969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:57.794973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:57.795158 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:57.795688 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:57.795696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:57.796038 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 14:58:57.796305 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 14:58:57.796823 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":259,"threads_started":1}
I20260925 14:58:57.799654 11396 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38093
I20260925 14:58:57.800000 11396 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:57.800709 11396 heartbeater.cc:511] Master 127.4.243.254:38093 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:57.802848 11207 ts_manager.cc:194] Registered new tserver with Master: b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633)
I20260925 14:58:57.804396 11207 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:47871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab.
Entry for principal oryx/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab.
I20260925 14:58:57.829106 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38093
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab
--principal=oryx/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:58.140928 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:58.141178 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:58.141263 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:58.148038 11402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:58.148132 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:58.148236 11402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:58:58.157248 11402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab
--principal=oryx/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:58.158360 11402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:58.159626 11402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:58.172510 11408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:58.172570 11407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:58.172595 11410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:58.173632 11402 server_base.cc:1034] running on GCE node
I20260925 14:58:58.174193 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:58.175143 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:58.176362 11402 hybrid_clock.cc:648] HybridClock initialized: now 1790348338176314 us; error 42 us; skew 500 ppm
I20260925 14:58:58.176661 11402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:58 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348338, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:58.179733 11402 init.cc:377] Logged in from keytab as oryx/127.4.243.194@KRBTEST.COM (short username oryx)
I20260925 14:58:58.181185 11402 webserver.cc:460] Webserver started at http://127.4.243.194:35245/ using document root <none> and password file <none>
I20260925 14:58:58.181816 11402 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:58.181910 11402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:58.182171 11402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:58.185048 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "f2e7fdf2d9774f679abcb8c82993ec2b"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1e7dde8b7c9acfc91ccc7058e2b3013"
server_key_iv: "598294ba1f3cfe569b82c1e7a9541c89"
server_key_version: "encryptionkey@0"
I20260925 14:58:58.185743 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "f2e7fdf2d9774f679abcb8c82993ec2b"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1e7dde8b7c9acfc91ccc7058e2b3013"
server_key_iv: "598294ba1f3cfe569b82c1e7a9541c89"
server_key_version: "encryptionkey@0"
I20260925 14:58:58.190800 11402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 14:58:58.194100 11417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:58.195524 11402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:58.195653 11402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "f2e7fdf2d9774f679abcb8c82993ec2b"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1e7dde8b7c9acfc91ccc7058e2b3013"
server_key_iv: "598294ba1f3cfe569b82c1e7a9541c89"
server_key_version: "encryptionkey@0"
I20260925 14:58:58.195830 11402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:58.225370 11402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:58.229667 11402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:58.229969 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:58.230935 11402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:58.232280 11402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:58.232347 11402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:58.232473 11402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:58.232527 11402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:58.264808 11402 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:46197
I20260925 14:58:58.265018 11529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:46197 every 8 connection(s)
I20260925 14:58:58.266130 11402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:58.266723 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11402
I20260925 14:58:58.266942 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:58.267511 5071 external_mini_cluster.cc:1428] Setting key dbcdf7c29de386d6bbe6ed2fa4011a39
Sep 25 14:58:58 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348338, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:58.288007 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:58.268824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:37049 (local address 127.4.243.254:38093)
0925 14:58:58.269023 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:58.269029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:58.270491 (+ 1462us) server_negotiation.cc:407] Connection header received
0925 14:58:58.272354 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:58.272362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:58.272452 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:58.272558 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:58.274344 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.275486 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.278031 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.278460 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.278553 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:58.282463 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:58.282469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:58.282472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:58.282501 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:58.285105 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:58.285795 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:58.285802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:58.285810 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:58.285928 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:58.286360 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:58.286371 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:58.286374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:58.286662 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:58.286813 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:58.287465 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 14:58:58.287671 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:58:58.289002 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:58.269527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.194:37049)
0925 14:58:58.270315 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:58.270369 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:58.272083 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:58.272812 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:58.272824 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:58.273490 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:58.274166 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:58.274189 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.275638 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.275646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:58.277886 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:58.277904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.278708 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.278718 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:58.279045 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:58.279723 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:58.279754 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:58.282252 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:58.285242 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:58.285251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:58.285267 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:58.285666 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:58.286055 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:58.286061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:58.286064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:58.286243 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:58.286786 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:58.286794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:58.287119 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 14:58:58.287376 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 14:58:58.287874 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":334,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.195@KRBTEST.COM" created.
I20260925 14:58:58.290506 11530 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38093
I20260925 14:58:58.290843 11530 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:58.291683 11530 heartbeater.cc:511] Master 127.4.243.254:38093 requested a full tablet report, sending...
I20260925 14:58:58.293148 11207 ts_manager.cc:194] Registered new tserver with Master: f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197)
I20260925 14:58:58.293881 11207 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:37049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab.
Entry for principal oryx/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab.
I20260925 14:58:58.313171 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38093
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab
--principal=oryx/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:58.618480 11535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:58.618799 11535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:58.618886 11535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:58.625451 11535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:58.625541 11535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:58.625639 11535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:58:58.634567 11535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab
--principal=oryx/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:58.635730 11535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:58.636929 11535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:58.649878 11540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:58.649924 11543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:58.649973 11541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:58.650700 11535 server_base.cc:1034] running on GCE node
I20260925 14:58:58.651229 11535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:58.652002 11535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:58.653219 11535 hybrid_clock.cc:648] HybridClock initialized: now 1790348338653185 us; error 48 us; skew 500 ppm
I20260925 14:58:58.653514 11535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:58 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348338, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:58.664186 11535 init.cc:377] Logged in from keytab as oryx/127.4.243.195@KRBTEST.COM (short username oryx)
I20260925 14:58:58.665747 11535 webserver.cc:460] Webserver started at http://127.4.243.195:42975/ using document root <none> and password file <none>
I20260925 14:58:58.666445 11535 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:58.666601 11535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:58.666939 11535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:58.669775 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "db0f5351dab84cf89d80a37e0b7be0ed"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1f7b01de59481a58a2f19ace389cc03"
server_key_iv: "69cd48f2444825234ad8124f13b5671f"
server_key_version: "encryptionkey@0"
I20260925 14:58:58.670526 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "db0f5351dab84cf89d80a37e0b7be0ed"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1f7b01de59481a58a2f19ace389cc03"
server_key_iv: "69cd48f2444825234ad8124f13b5671f"
server_key_version: "encryptionkey@0"
I20260925 14:58:58.675669 11535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 14:58:58.679467 11550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:58.681365 11535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:58:58.681540 11535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "db0f5351dab84cf89d80a37e0b7be0ed"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1f7b01de59481a58a2f19ace389cc03"
server_key_iv: "69cd48f2444825234ad8124f13b5671f"
server_key_version: "encryptionkey@0"
I20260925 14:58:58.681792 11535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:58.731904 11535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:58.735844 11535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:58.736061 11535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:58.736865 11535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:58.738091 11535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:58.738165 11535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:58.738243 11535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:58.738286 11535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:58.770134 11535 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:41717
I20260925 14:58:58.770175 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:41717 every 8 connection(s)
I20260925 14:58:58.771392 11535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:58.780067 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11535
I20260925 14:58:58.780272 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:58.780690 5071 external_mini_cluster.cc:1428] Setting key dbdd9a37cfbeab8fa0053386c9a3e629
Sep 25 14:58:58 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348338, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:58:58.790942 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:58.773793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:49637 (local address 127.4.243.254:38093)
0925 14:58:58.773941 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:58.773944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:58.775218 (+ 1274us) server_negotiation.cc:407] Connection header received
0925 14:58:58.776937 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:58.776942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:58.777034 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:58.777140 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:58.778582 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.779323 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.780939 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.781449 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.781552 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:58.784925 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:58.784930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:58.784933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:58.784962 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:58.787341 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:58.788161 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:58.788166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:58.788170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:58.788338 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:58.788845 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:58.788850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:58.788853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:58.789255 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:58.789398 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:58.790211 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 14:58:58.790494 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:58:58.791791 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:58.774374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.195:49637)
0925 14:58:58.775070 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:58.775104 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:58.776643 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:58.777352 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:58.777360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:58.777912 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:58.778420 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:58.778435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.779476 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.779486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:58.780723 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:58.780734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:58.781654 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:58.781660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:58.781890 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:58.782642 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:58.782670 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:58.784745 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:58.787479 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:58.787485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:58.787496 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:58.787970 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:58.788445 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:58.788449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:58.788453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:58.788680 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:58.789374 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:58.789382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:58.789813 (+ 431us) client_negotiation.cc:769] Sending connection context
0925 14:58:58.790124 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 14:58:58.790627 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":306,"threads_started":1}
I20260925 14:58:58.792977 11663 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38093
I20260925 14:58:58.793262 11663 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:58.793972 11663 heartbeater.cc:511] Master 127.4.243.254:38093 requested a full tablet report, sending...
I20260925 14:58:58.795616 11207 ts_manager.cc:194] Registered new tserver with Master: db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:58:58.796384 11207 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:49637
I20260925 14:58:58.807966 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:58.808151 11396 heartbeater.cc:503] Master 127.4.243.254:38093 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.4.243.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.253.keytab.
Entry for principal oryx/127.4.243.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.253.keytab.
I20260925 14:58:58.850531 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:37649
--webserver_interface=127.4.243.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.253.keytab
--principal=oryx/127.4.243.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:38093,127.4.243.253:37649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.167977 11668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.168258 11668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.168417 11668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.175711 11668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:59.175817 11668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.175967 11668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.176043 11668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:59.176079 11668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:59.185768 11668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:38093,127.4.243.253:37649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.253.keytab
--principal=oryx/127.4.243.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:37649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.243.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
W20260925 14:58:59.186138 11668 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.243.254:38093,127.4.243.253:37649'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 14:58:59.187242 11668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.188578 11668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.199283 11673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.199308 11676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.199250 11674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.200249 11668 server_base.cc:1034] running on GCE node
I20260925 14:58:59.200875 11668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.202060 11668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.203259 11668 hybrid_clock.cc:648] HybridClock initialized: now 1790348339203197 us; error 36 us; skew 500 ppm
I20260925 14:58:59.203516 11668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.206517 11668 init.cc:377] Logged in from keytab as oryx/127.4.243.253@KRBTEST.COM (short username oryx)
I20260925 14:58:59.207844 11668 webserver.cc:460] Webserver started at http://127.4.243.253:43405/ using document root <none> and password file <none>
I20260925 14:58:59.208431 11668 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.208509 11668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.208747 11668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.211478 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/instance:
uuid: "2492809c32e7400588ed011c799620e5"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "8639b02a6ec8b9e46f2ed82c781bce14"
server_key_iv: "2b2f2d3f36748a637a7888775ff61c9f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.212165 11668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal/instance:
uuid: "2492809c32e7400588ed011c799620e5"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "8639b02a6ec8b9e46f2ed82c781bce14"
server_key_iv: "2b2f2d3f36748a637a7888775ff61c9f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.216701 11668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:58:59.219750 11683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.220995 11668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:59.221169 11668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
uuid: "2492809c32e7400588ed011c799620e5"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "8639b02a6ec8b9e46f2ed82c781bce14"
server_key_iv: "2b2f2d3f36748a637a7888775ff61c9f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.221380 11668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.235234 11668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.239079 11668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.239322 11668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.256670 11668 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.253:37649
I20260925 14:58:59.256688 11734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.253:37649 every 8 connection(s)
I20260925 14:58:59.257930 11668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
I20260925 14:58:59.261392 11735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.267808 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11668
I20260925 14:58:59.267529 11735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.253@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.253@KRBTEST.COM
I20260925 14:58:59.287402 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.270989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.4.243.254:38093)
0925 14:58:59.271141 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.271145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.271865 (+ 720us) server_negotiation.cc:407] Connection header received
0925 14:58:59.273164 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.273167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.273227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.273299 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.274611 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.275306 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.276961 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.277647 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.277746 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.281092 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.281097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.281100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.281128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.284274 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.285099 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.285107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.285110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.285246 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.285680 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.285685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.285688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.285981 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.286147 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.286819 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.287082 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:58:59.288460 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.269937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.4.243.253:37649)
0925 14:58:59.270577 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.270641 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.270694 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:58:59.272913 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.272937 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.273202 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.273409 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.274734 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.276095 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.277520 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.278230 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.278376 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.282310 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.282318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.282335 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.282361 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.285280 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.285913 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.285924 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.285929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.286073 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.286429 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.286435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.286439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.286822 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.287175 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.287776 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.287974 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":298,"threads_started":1}
I20260925 14:58:59.288547 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.270914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37008)
0925 14:58:59.271716 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.271758 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.272913 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.273381 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.273386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.274127 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.274429 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.274444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.275439 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.275443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.276812 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.276823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.277888 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.277896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.278146 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.278936 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.278955 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.280915 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.284412 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.284422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.284433 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.284961 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.285361 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.285366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.285369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.285554 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.286129 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.286136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.286436 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.286759 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.287215 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":378,"threads_started":1}
I20260925 14:58:59.291123 11735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:58:59.296566 11530 heartbeater.cc:503] Master 127.4.243.254:38093 was elected leader, sending a full tablet report...
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.253@KRBTEST.COM for oryx/127.4.243.253@KRBTEST.COM
I20260925 14:58:59.302461 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.291917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54574)
0925 14:58:59.292125 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.292139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.292216 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.292520 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.292523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.292796 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.292918 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.292924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.293836 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.293839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.294728 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.294738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.295351 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.295354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.295478 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.296013 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.296029 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.297968 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.300661 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.300664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.300666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.301024 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.301299 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.301302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.301304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.301413 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.301810 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.301813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.301956 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.302059 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.302235 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:58:59.302700 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.291962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.4.243.253:37649)
0925 14:58:59.292172 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.292178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.292195 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:58:59.292323 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.292328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.292402 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.292501 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.293054 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.293719 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.294842 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.295234 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.295330 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.298148 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.298152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.298155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.298180 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.300553 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.301116 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.301119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.301121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.301213 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.301495 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.301498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.301500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.301722 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.301816 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.302131 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.302355 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":83}
I20260925 14:58:59.307011 11735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Bootstrap starting.
I20260925 14:58:59.309777 11735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.310768 11735 log.cc:824] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.313021 11735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: No bootstrap required, opened a new log
I20260925 14:58:59.319437 11735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } }
I20260925 14:58:59.319794 11735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.319885 11735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2492809c32e7400588ed011c799620e5, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.320355 11735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } }
I20260925 14:58:59.321076 11741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } } }
I20260925 14:58:59.321426 11741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:58:59.322011 11735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:58:59.330195 11752 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:59.330348 11752 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:59.332685 11668 master_runner.cc:428] Detected that this master 2492809c32e7400588ed011c799620e5 is joining an existing cluster
I20260925 14:58:59.332762 11668 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.243.253:37649
I20260925 14:58:59.352474 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.336379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.4.243.253:37649)
0925 14:58:59.336614 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.336620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.337665 (+ 1045us) server_negotiation.cc:407] Connection header received
0925 14:58:59.337813 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.337818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.337874 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.337980 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.339392 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.340074 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.342127 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.342856 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.342934 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.345745 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.345750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.345755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.345786 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.349595 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.350410 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.350415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.350419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.350553 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.350994 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.350998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.351001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.351321 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.351450 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.351937 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.352144 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 14:58:59.352664 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.336279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.4.243.254:38093)
0925 14:58:59.336450 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.336456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.337302 (+ 846us) server_negotiation.cc:407] Connection header received
0925 14:58:59.337632 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.337639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.337723 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.337823 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.339249 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.340313 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.342138 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.342805 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.342901 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.345755 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.345760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.345765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.345800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.349857 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.350587 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.350615 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.350621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.350765 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.351162 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.351166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.351169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.351486 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.351586 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.352117 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.352317 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:58:59.353403 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.336751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54578)
0925 14:58:59.337600 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.337614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.337716 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.338145 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.338151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.338852 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.339284 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.339291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.340229 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.340234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.342001 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.342008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.343197 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.343201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.343420 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.344233 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.344261 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.345578 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.349728 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.349732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.349736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.350301 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.350716 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.350719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.350723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.350874 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.351533 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.351538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.351675 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.352686 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.352947 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":93,"thread_start_us":512,"threads_started":1}
I20260925 14:58:59.353559 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.336513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37014)
0925 14:58:59.337194 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.337219 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.337411 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.337850 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.337856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.338583 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.339084 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.339096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.340441 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.340445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.341995 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.342008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.343173 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.343183 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.343423 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.344312 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.344334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.345578 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.350003 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.350013 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.350018 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.350460 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.350894 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.350899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.350902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.351042 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.351659 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.351663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.351836 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.352852 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.353127 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":112,"spinlock_wait_cycles":40448,"thread_start_us":420,"threads_started":1}
WARNING: no policy specified for oryx/127.4.243.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.252@KRBTEST.COM" created.
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348337, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.254@KRBTEST.COM for oryx/127.4.243.253@KRBTEST.COM
I20260925 14:58:59.380913 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.358719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.4.243.253:37649)
0925 14:58:59.358881 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.358889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.363221 (+ 4332us) server_negotiation.cc:407] Connection header received
0925 14:58:59.363528 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.363538 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.363592 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.363704 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.364629 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.365318 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.366573 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.367120 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.367254 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.375186 (+ 7932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.375192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.375197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.375235 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.378316 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.379059 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.379065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.379069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.379187 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.379548 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.379552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.379555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.379788 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.379941 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.380358 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.380554 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:58:59.380955 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.358722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54584)
0925 14:58:59.363097 (+ 4375us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.363128 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.363335 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.363745 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.363752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.364296 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.364485 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.364493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.365450 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.365456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.366447 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.366457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.367287 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.367292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.367487 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.368177 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.368193 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.374931 (+ 6738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.378451 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.378462 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.378469 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.378946 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.379300 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.379306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.379309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.379441 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.379914 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.379922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.380100 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.380268 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.380554 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4195,"thread_start_us":4044,"threads_started":1}
I20260925 14:58:59.382514 11207 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.243.253:37649: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2492809c32e7400588ed011c799620e5"
member_type: NON_VOTER
last_known_addr {
host: "127.4.243.253"
port: 37649
}
attrs {
promote: true
}
}
dest_uuid: "6b4bf6ce139749de929c37851c302282"
cas_config_opid_index: -1
I20260925 14:58:59.383504 11207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } }
I20260925 14:58:59.390331 11764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2492809c32e7400588ed011c799620e5 (127.4.243.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } }
I20260925 14:58:59.391423 11710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.394155 11764 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.243.253:37649
I20260925 14:58:59.394826 11668 master.cc:537] Master@127.4.243.253:37649 shutting down...
I20260925 14:58:59.396279 11710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Refusing update from remote peer 6b4bf6ce139749de929c37851c302282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 14:58:59.396318 11765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } }
I20260925 14:58:59.396678 11765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:59.397859 11767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:58:59.401679 11192 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.252.keytab.
Entry for principal oryx/127.4.243.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.252.keytab.
I20260925 14:58:59.404093 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:35727
--webserver_interface=127.4.243.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.252.keytab
--principal=oryx/127.4.243.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:59.406140 11668 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:59.406505 11668 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:59.406718 11668 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: stopping tablet replica
I20260925 14:58:59.411065 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.410242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54600)
0925 14:58:59.410770 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.410901 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
W20260925 14:58:59.411422 11192 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 -> Peer 2492809c32e7400588ed011c799620e5 (127.4.243.253:37649): Couldn't send request to peer 2492809c32e7400588ed011c799620e5. Status: Network error: Client connection negotiation failed: client connection to 127.4.243.253:37649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:58:59.413383 11668 master.cc:552] Master@127.4.243.253:37649 shutdown complete.
I20260925 14:58:59.413466 11668 master_runner.cc:305] Clearing existing system tablet
I20260925 14:58:59.413898 11668 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.413992 11668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.417073 11668 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 14:58:59.418807 11771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.419194 11668 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.419622 11668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 14:58:59.419761 11668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
uuid: "2492809c32e7400588ed011c799620e5"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "8639b02a6ec8b9e46f2ed82c781bce14"
server_key_iv: "2b2f2d3f36748a637a7888775ff61c9f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.425753 11668 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:59.432775 11668 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:58:59.432999 11668 log.cc:1196] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 14:58:59.433548 11668 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Deleting consensus metadata
I20260925 14:58:59.433830 11668 master_runner.cc:315] Copying system tablet from 127.4.243.254:38093
I20260925 14:58:59.436157 11668 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: tablet copy: Beginning tablet copy session from remote peer at address 127.4.243.254:38093
I20260925 14:58:59.450855 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.436954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.4.243.254:38093)
0925 14:58:59.437182 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.437188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.437475 (+ 287us) server_negotiation.cc:407] Connection header received
0925 14:58:59.437670 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.437675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.437765 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.437875 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.438915 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.439980 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.441687 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.442339 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.442475 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.444535 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.444541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.444546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.444584 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.448012 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.448754 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.448759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.448764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.448898 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.449280 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.449285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.449288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.449655 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.449813 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.450255 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.450488 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 14:58:59.450861 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.436894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37024)
0925 14:58:59.437363 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.437382 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.437539 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.437918 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.437923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.438444 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.438734 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.438747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.440172 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.440178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.441551 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.441563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.442513 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.442519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.442696 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.443391 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.443411 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.444322 (+ 911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.448146 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.448151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.448155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.448631 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.449013 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.449018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.449021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.449169 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.449787 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.449792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.449970 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.450133 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.450414 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"spinlock_wait_cycles":3968,"thread_start_us":296,"threads_started":1}
I20260925 14:58:59.452085 11227 tablet_copy_service.cc:140] P 6b4bf6ce139749de929c37851c302282: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2492809c32e7400588ed011c799620e5 ({username='oryx', principal='oryx/127.4.243.253@KRBTEST.COM'} at 127.0.0.1:37024)
I20260925 14:58:59.452412 11227 tablet_copy_service.cc:161] P 6b4bf6ce139749de929c37851c302282: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2492809c32e7400588ed011c799620e5 at {username='oryx', principal='oryx/127.4.243.253@KRBTEST.COM'} at 127.0.0.1:37024: session id = 2492809c32e7400588ed011c799620e5-00000000000000000000000000000000
I20260925 14:58:59.456830 11227 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:58:59.458937 11668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:58:59.467016 11668 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: tablet copy: Starting download of 0 data blocks...
I20260925 14:58:59.467299 11668 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: tablet copy: Starting download of 1 WAL segments...
I20260925 14:58:59.469909 11668 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:58:59.474737 11227 tablet_copy_service.cc:342] P 6b4bf6ce139749de929c37851c302282: Request end of tablet copy session 2492809c32e7400588ed011c799620e5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.243.253@KRBTEST.COM'} at 127.0.0.1:37024
I20260925 14:58:59.474957 11227 tablet_copy_service.cc:433] P 6b4bf6ce139749de929c37851c302282: ending tablet copy session 2492809c32e7400588ed011c799620e5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2492809c32e7400588ed011c799620e5
I20260925 14:58:59.485369 11668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.502997 11778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.506850 11779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.513660 11668 server_base.cc:1034] running on GCE node
W20260925 14:58:59.514537 11781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.514974 11668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.515606 11668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.516811 11668 hybrid_clock.cc:648] HybridClock initialized: now 1790348339516752 us; error 36 us; skew 500 ppm
I20260925 14:58:59.517069 11668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.519385 11668 init.cc:377] Logged in from keytab as oryx/127.4.243.253@KRBTEST.COM (short username oryx)
I20260925 14:58:59.533450 11668 webserver.cc:460] Webserver started at http://127.4.243.253:33075/ using document root <none> and password file <none>
I20260925 14:58:59.533818 11668 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.533934 11668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.536995 11668 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 14:58:59.538807 11788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.539259 11668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:58:59.539446 11668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
uuid: "2492809c32e7400588ed011c799620e5"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "8639b02a6ec8b9e46f2ed82c781bce14"
server_key_iv: "2b2f2d3f36748a637a7888775ff61c9f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.539685 11668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.563952 11668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.568426 11668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.648519 11668 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.253:37649
I20260925 14:58:59.648563 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.253:37649 every 8 connection(s)
I20260925 14:58:59.649714 11668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
I20260925 14:58:59.656978 11840 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:58:59.660948 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Bootstrap starting.
I20260925 14:58:59.678742 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:59.679268 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Bootstrap complete.
I20260925 14:58:59.680262 11840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } }
I20260925 14:58:59.680516 11840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2492809c32e7400588ed011c799620e5, State: Initialized, Role: LEARNER
I20260925 14:58:59.680796 11840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } }
I20260925 14:58:59.681612 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } }
I20260925 14:58:59.681860 11840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:59.681954 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:58:59.695302 11854 catalog_manager.cc:1259] Loaded cluster ID: f8d2a6c02ade41339d86bfab51123204
I20260925 14:58:59.695412 11854 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: loading cluster ID for follower catalog manager: success
I20260925 14:58:59.699376 11854 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: acquiring CA information for follower catalog manager: success
I20260925 14:58:59.701891 11854 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:58:59.798738 11663 heartbeater.cc:503] Master 127.4.243.254:38093 was elected leader, sending a full tablet report...
W20260925 14:58:59.807565 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.807891 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.808063 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.816102 11769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:59.816237 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.816341 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.816433 11769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:59.816509 11769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:59.826001 11769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.252.keytab
--principal=oryx/127.4.243.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.243.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:58:59.827785 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.829540 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:59.834717 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.827855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54606)
0925 14:58:59.828137 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.828163 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.828301 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.829298 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.829305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.829335 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.829520 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.829533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.831469 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.831478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.832652 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.832664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.833853 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.833859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.834090 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.834141 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.834249 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.834396 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 14:58:59.834774 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.828093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.4.243.253:37649)
0925 14:58:59.828932 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.828938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.828958 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:58:59.829031 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.829039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.829121 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.829238 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.829758 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.831319 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.832794 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.833673 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.833836 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.834063 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.834270 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.834396 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":534,"threads_started":1}
I20260925 14:58:59.836261 11815 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 14:58:59.836874 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4bf6ce139749de929c37851c302282. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } }
I20260925 14:58:59.837142 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: LEARNER
W20260925 14:58:59.844431 11861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.844619 11860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.844977 11863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.845643 11769 server_base.cc:1034] running on GCE node
I20260925 14:58:59.846429 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.847846 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.849067 11769 hybrid_clock.cc:648] HybridClock initialized: now 1790348339849023 us; error 39 us; skew 500 ppm
I20260925 14:58:59.849449 11769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.852586 11769 init.cc:377] Logged in from keytab as oryx/127.4.243.252@KRBTEST.COM (short username oryx)
I20260925 14:58:59.854136 11769 webserver.cc:460] Webserver started at http://127.4.243.252:32785/ using document root <none> and password file <none>
I20260925 14:58:59.854863 11769 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.854986 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.855286 11769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.858084 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/instance:
uuid: "4f029608a0734b91877cda219816e8f4"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "9c1ef3d8912d8570f22814593a4922f4"
server_key_iv: "e1589ba02aa798005943d94561077689"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.859288 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal/instance:
uuid: "4f029608a0734b91877cda219816e8f4"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "9c1ef3d8912d8570f22814593a4922f4"
server_key_iv: "e1589ba02aa798005943d94561077689"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.864279 11769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:58:59.867991 11870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.869603 11769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:59.869766 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
uuid: "4f029608a0734b91877cda219816e8f4"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "9c1ef3d8912d8570f22814593a4922f4"
server_key_iv: "e1589ba02aa798005943d94561077689"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.869963 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.884306 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.888285 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.888562 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.906539 11769 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.252:35727
I20260925 14:58:59.906559 11921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.252:35727 every 8 connection(s)
I20260925 14:58:59.907872 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
I20260925 14:58:59.910704 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11769
I20260925 14:58:59.911473 11922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.919886 11922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.252@KRBTEST.COM
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.252@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:58:59.936650 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.912662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.4.243.252:35727)
0925 14:58:59.913395 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.913406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.913442 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:58:59.915004 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.915024 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.915399 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.915775 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.917156 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.918574 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.920350 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.922099 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.922312 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.928288 (+ 5976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.928298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.928313 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.928357 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.932339 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.933087 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.933093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.933099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.933235 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.933651 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.933655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.933657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.934158 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.934455 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.934980 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.935714 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":375,"threads_started":1}
I20260925 14:58:59.939603 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.923378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37026)
0925 14:58:59.924488 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.924531 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.924741 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.925381 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.925391 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.926200 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.926518 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.926531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.927981 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.927986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.929713 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.929739 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.930633 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.930640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.930834 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.931379 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.931403 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.934091 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.937052 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.937063 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.937070 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.937483 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.937870 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.937873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.937875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.938008 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.938646 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.938655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.938800 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.938960 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.939237 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":726,"threads_started":1}
I20260925 14:58:59.939636 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.923426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.4.243.254:38093)
0925 14:58:59.923636 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.923641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.924622 (+ 981us) server_negotiation.cc:407] Connection header received
0925 14:58:59.924961 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.924968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.925060 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.925165 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.926702 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.927798 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.929870 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.930419 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.930513 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.934263 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.934268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.934271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.934299 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.936927 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.937603 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.937608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.937612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.937760 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.938126 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.938133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.938136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.938463 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.938643 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.939120 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.939345 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 14:58:59.940799 11922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.252@KRBTEST.COM for oryx/127.4.243.253@KRBTEST.COM
I20260925 14:58:59.951865 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.941507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54612)
0925 14:58:59.941704 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.941727 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.941798 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.942047 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.942050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.942306 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.942420 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.942427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.943384 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.943388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.944216 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.944225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.944777 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.944781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.944920 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.945330 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.945344 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.947149 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.949682 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.949688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.949692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.950037 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.950393 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.950399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.950402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.950528 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.951012 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.951016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.951149 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.951282 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.951548 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 14:58:59.951889 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.941610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.4.243.253:37649)
0925 14:58:59.941759 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.941764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.941780 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:58:59.941875 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.941879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.941934 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.942019 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.942528 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.943279 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.944321 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.944666 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.944747 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.947330 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.947336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.947339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.947366 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.949560 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.950152 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.950159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.950162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.950287 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.950655 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.950660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.950663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.950902 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.951044 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.951360 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.951562 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:58:59.952850 11922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:59 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.252@KRBTEST.COM for oryx/127.4.243.252@KRBTEST.COM
I20260925 14:58:59.965090 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.953496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.0.0.1:60716)
0925 14:58:59.953639 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.953658 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.953761 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.954066 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.954069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.954371 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.954504 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.954511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.955484 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.955488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.956280 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.956290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.957307 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.957311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.957480 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.957963 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.957978 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.959879 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.963144 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.963148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.963150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.963490 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.963811 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.963814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.963816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.963951 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.964405 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.964408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.964519 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.964624 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.964784 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 14:58:59.965237 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.953555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.4.243.252:35727)
0925 14:58:59.953693 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.953697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.953730 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:58:59.953896 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.953900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.953956 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.954036 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.954688 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.955340 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.956399 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.957182 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.957272 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.960026 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.960030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.960033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.960059 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.963012 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.963595 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.963598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.963600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.963721 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.964039 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.964041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.964043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.964303 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.964409 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.964703 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.964933 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":49}
I20260925 14:58:59.970093 11922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Bootstrap starting.
I20260925 14:58:59.973114 11922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.974207 11922 log.cc:824] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.976605 11922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: No bootstrap required, opened a new log
I20260925 14:58:59.983510 11922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } }
I20260925 14:58:59.983793 11922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.983913 11922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f029608a0734b91877cda219816e8f4, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.984474 11922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } }
I20260925 14:58:59.985286 11928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } } }
I20260925 14:58:59.985560 11928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:58:59.986303 11922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:58:59.995018 11939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:59.995216 11939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:59.999676 11769 master_runner.cc:428] Detected that this master 4f029608a0734b91877cda219816e8f4 is joining an existing cluster
I20260925 14:58:59.999797 11769 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.243.252:35727
I20260925 14:59:00.023084 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.004117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.4.243.253:37649)
0925 14:59:00.004306 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.004314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.005115 (+ 801us) server_negotiation.cc:407] Connection header received
0925 14:59:00.005423 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.005432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.005531 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.005617 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.007537 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.009005 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.012219 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.013173 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.013294 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.016124 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.016131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.016136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.016175 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.019665 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.020410 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.020415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.020418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.020559 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.020984 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.020990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.020993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.021316 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.021447 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.022054 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.022314 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:59:00.024019 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.004125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54620)
0925 14:59:00.004989 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.005015 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.005257 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.005902 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.005910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.006852 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.007311 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.007328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.009232 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.009241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.012019 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.012037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.013651 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.013660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.013913 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.014829 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.014856 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.015934 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.019824 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.019829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.019834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.020279 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.020682 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.020686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.020690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.020864 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.021513 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.021522 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.021721 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.023274 (+ 1553us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.023558 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":457,"threads_started":1}
I20260925 14:59:00.031122 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.004328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37032)
0925 14:59:00.015278 (+ 10950us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.015303 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.015465 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.015989 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.015997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.016654 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.017010 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.017043 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.018633 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.018639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.020439 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.020451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.021547 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.021558 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.021714 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.022314 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.022334 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.024401 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.028107 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.028112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.028116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.028616 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.029041 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.029046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.029049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.029231 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.029915 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.029919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.030179 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.030335 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.030665 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10744,"spinlock_wait_cycles":25856,"thread_start_us":10471,"threads_started":1}
I20260925 14:59:00.031801 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.004001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.4.243.254:38093)
0925 14:59:00.004193 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.004201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.015432 (+ 11231us) server_negotiation.cc:407] Connection header received
0925 14:59:00.015617 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.015623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.015702 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.015807 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.017226 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.018406 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.020579 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.021203 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.021288 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.024600 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.024607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.024610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.024644 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.027976 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.028753 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.028759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.028764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.028925 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.029386 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.029392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.029395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.029720 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.029842 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.030493 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.030717 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:59:00.037832 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.005704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.0.0.1:60732)
0925 14:59:00.023666 (+ 17962us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.023680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.023821 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.024344 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.024358 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.024976 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.025240 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.025251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.026640 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.026648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.028288 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.028324 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.029277 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.029283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.029446 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.029992 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.030008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.031032 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.035061 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.035068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.035072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.035503 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.035888 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.035894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.035899 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.036042 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.036728 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.036735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.036872 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.036995 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.037278 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17823,"thread_start_us":17693,"threads_started":1}
I20260925 14:59:00.038815 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.005269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.4.243.252:35727)
0925 14:59:00.005425 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.005431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.023762 (+ 18331us) server_negotiation.cc:407] Connection header received
0925 14:59:00.023954 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.023958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.024020 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.024100 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.025402 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.026455 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.028470 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.029093 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.029237 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.031635 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.031644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.031650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.031700 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.034934 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.035622 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.035626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.035630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.035772 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.036177 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.036183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.036194 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.036547 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.036684 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.038194 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.038400 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:59:00.042770 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11268
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348337, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.254@KRBTEST.COM for oryx/127.4.243.252@KRBTEST.COM
I20260925 14:59:00.053357 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.036801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.0.0.1:60740)
0925 14:59:00.036994 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.037013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.037100 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.038083 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.038089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.038484 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.038720 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.038735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.040446 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.040450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.041591 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.041603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.042917 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.042925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.043111 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.043738 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.043753 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.046426 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.050213 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.050219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.050224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.050762 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.051121 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.051125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.051129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.051257 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.052389 (+ 1132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.052394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.052534 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.052653 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.052968 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:59:00.054543 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.037077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.4.243.252:35727)
0925 14:59:00.037678 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.037684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.037704 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:59:00.037792 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.037798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.037900 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.038672 (+ 772us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.039054 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.040330 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.041765 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.042725 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.043458 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.046691 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.046697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.046701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.046733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.050062 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.050894 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.050897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.050899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.051012 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.051364 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.051370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.051373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.052235 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.053627 (+ 1392us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.053768 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.054010 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":273,"threads_started":1}
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.193@KRBTEST.COM
I20260925 14:59:00.056551 11207 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.243.252:35727: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4f029608a0734b91877cda219816e8f4"
member_type: NON_VOTER
last_known_addr {
host: "127.4.243.252"
port: 35727
}
attrs {
promote: true
}
}
dest_uuid: "6b4bf6ce139749de929c37851c302282"
cas_config_opid_index: 5
I20260925 14:59:00.057569 11207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } }
I20260925 14:59:00.059341 11767 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4f029608a0734b91877cda219816e8f4 (127.4.243.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.063143 11815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 LEARNER]: Refusing update from remote peer 6b4bf6ce139749de929c37851c302282: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:59:00.063651 11897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:00.064385 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.044130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.4.243.193:39633)
0925 14:59:00.044749 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.044759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.044789 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:59:00.044869 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.044877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.045164 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.045387 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.047078 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.048658 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.050529 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.051448 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.051580 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.057083 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.057095 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.057109 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.057141 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.060823 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.061463 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.061472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.061476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.061601 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.061986 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.061991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.061995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.062331 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.062644 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.063443 (+ 799us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.063707 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":301,"threads_started":1}
I20260925 14:59:00.066854 11897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Refusing update from remote peer 6b4bf6ce139749de929c37851c302282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:59:00.066861 11767 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.243.252:35727
I20260925 14:59:00.067075 11951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 14:59:00.067416 11769 master.cc:537] Master@127.4.243.252:35727 shutting down...
I20260925 14:59:00.067798 11391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:00.075453 11815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4f029608a0734b91877cda219816e8f4 (127.4.243.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.081244 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.081516 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:59:00.089423 11954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 14:59:00.094691 11767 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: attempting to promote NON_VOTER 2492809c32e7400588ed011c799620e5 to VOTER
I20260925 14:59:00.095003 11951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: NON_VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: true } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.095319 11951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:00.095667 11767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } }
I20260925 14:59:00.098028 11815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 LEARNER]: Refusing update from remote peer 6b4bf6ce139749de929c37851c302282: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 14:59:00.101109 11769 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:00.101218 11950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 14:59:00.101496 11769 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:00.101718 11769 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: stopping tablet replica
I20260925 14:59:00.106770 11955 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2492809c32e7400588ed011c799620e5 (127.4.243.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.107386 11815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2492809c32e7400588ed011c799620e5 (127.4.243.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.108670 11769 master.cc:552] Master@127.4.243.252:35727 shutdown complete.
I20260925 14:59:00.108758 11769 master_runner.cc:305] Clearing existing system tablet
I20260925 14:59:00.109118 11769 fs_manager.cc:359] Metadata directory not provided
W20260925 14:59:00.109084 11193 connection.cc:698] client connection to 127.4.243.252:35727 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260925 14:59:00.109200 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260925 14:59:00.109323 11193 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 -> Peer 4f029608a0734b91877cda219816e8f4 (127.4.243.252:35727): Couldn't send request to peer 4f029608a0734b91877cda219816e8f4. Status: Network error: sendmsg error: Broken pipe (error 32). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:59:00.111182 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.111426 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.110977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.0.0.1:60756)
0925 14:59:00.111162 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.111254 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 14:59:00.111449 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:00.113633 11951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.113947 11951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:00.114298 11955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.114657 11955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:00.115409 11769 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 14:59:00.116974 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.117281 11769 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:00.117568 11769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:59:00.117666 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
uuid: "4f029608a0734b91877cda219816e8f4"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "9c1ef3d8912d8570f22814593a4922f4"
server_key_iv: "e1589ba02aa798005943d94561077689"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.122293 11769 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:00.128894 11769 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:59:00.129048 11769 log.cc:1196] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 14:59:00.129513 11769 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Deleting consensus metadata
I20260925 14:59:00.129750 11769 master_runner.cc:315] Copying system tablet from 127.4.243.254:38093
I20260925 14:59:00.132660 11769 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: tablet copy: Beginning tablet copy session from remote peer at address 127.4.243.254:38093
I20260925 14:59:00.147558 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.133501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.4.243.254:38093)
0925 14:59:00.133739 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.133746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.134219 (+ 473us) server_negotiation.cc:407] Connection header received
0925 14:59:00.134396 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.134402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.134493 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.134615 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.135583 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.136705 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.138495 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.139140 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.139254 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.141437 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.141443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.141448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.141486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.144774 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.145541 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.145546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.145550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.145685 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.146071 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.146075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.146079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.146394 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.146533 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.146975 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.147185 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 14:59:00.148079 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.133414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37044)
0925 14:59:00.134107 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.134126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.134269 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.134662 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.134667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.135178 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.135429 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.135440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.136856 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.136862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.138340 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.138362 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.139286 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.139292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.139466 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.140142 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.140164 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.141232 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.144918 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.144924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.144928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.145424 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.145796 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.145801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.145804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.145961 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.146508 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.146513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.146699 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.146867 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.147162 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":437,"threads_started":1}
I20260925 14:59:00.148689 11227 tablet_copy_service.cc:140] P 6b4bf6ce139749de929c37851c302282: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4f029608a0734b91877cda219816e8f4 ({username='oryx', principal='oryx/127.4.243.252@KRBTEST.COM'} at 127.0.0.1:37044)
I20260925 14:59:00.148852 11227 tablet_copy_service.cc:161] P 6b4bf6ce139749de929c37851c302282: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4f029608a0734b91877cda219816e8f4 at {username='oryx', principal='oryx/127.4.243.252@KRBTEST.COM'} at 127.0.0.1:37044: session id = 4f029608a0734b91877cda219816e8f4-00000000000000000000000000000000
I20260925 14:59:00.152541 11227 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:59:00.154320 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:59:00.160852 11769 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: tablet copy: Starting download of 0 data blocks...
I20260925 14:59:00.161119 11769 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: tablet copy: Starting download of 1 WAL segments...
I20260925 14:59:00.163856 11769 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:59:00.167994 11227 tablet_copy_service.cc:342] P 6b4bf6ce139749de929c37851c302282: Request end of tablet copy session 4f029608a0734b91877cda219816e8f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.243.252@KRBTEST.COM'} at 127.0.0.1:37044
I20260925 14:59:00.168125 11227 tablet_copy_service.cc:433] P 6b4bf6ce139749de929c37851c302282: ending tablet copy session 4f029608a0734b91877cda219816e8f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4f029608a0734b91877cda219816e8f4
I20260925 14:59:00.180109 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:00.182732 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.187834 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:00.197957 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11268
I20260925 14:59:00.199075 11769 server_base.cc:1034] running on GCE node
W20260925 14:59:00.203609 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:00.204109 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:00.204766 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:00.206002 11769 hybrid_clock.cc:648] HybridClock initialized: now 1790348340205963 us; error 40 us; skew 500 ppm
I20260925 14:59:00.206290 11769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348340, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:00.208598 11769 init.cc:377] Logged in from keytab as oryx/127.4.243.252@KRBTEST.COM (short username oryx)
I20260925 14:59:00.211849 11769 webserver.cc:460] Webserver started at http://127.4.243.252:40457/ using document root <none> and password file <none>
I20260925 14:59:00.212247 11769 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:00.212364 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:00.215745 11769 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 14:59:00.217778 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.218225 11769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:59:00.218379 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
uuid: "4f029608a0734b91877cda219816e8f4"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-qxh0"
server_key: "9c1ef3d8912d8570f22814593a4922f4"
server_key_iv: "e1589ba02aa798005943d94561077689"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.218827 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:00.220446 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:39633
--local_ip_for_outbound_sockets=127.4.243.193
--tserver_master_addrs=127.4.243.254:38093
--webserver_port=37515
--webserver_interface=127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab
--principal=oryx/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:59:00.236971 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:00.241052 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:00.283316 11769 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.252:35727
I20260925 14:59:00.283493 12028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.252:35727 every 8 connection(s)
I20260925 14:59:00.284332 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
I20260925 14:59:00.290238 12029 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:59:00.293958 12029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Bootstrap starting.
I20260925 14:59:00.311167 12029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:59:00.311646 12029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: Bootstrap complete.
I20260925 14:59:00.312651 12029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } }
I20260925 14:59:00.312978 12029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f029608a0734b91877cda219816e8f4, State: Initialized, Role: LEARNER
I20260925 14:59:00.313212 12029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } }
I20260925 14:59:00.313907 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.314286 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:59:00.314291 12029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:00.321447 12043 catalog_manager.cc:1259] Loaded cluster ID: f8d2a6c02ade41339d86bfab51123204
I20260925 14:59:00.321593 12043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: loading cluster ID for follower catalog manager: success
I20260925 14:59:00.324877 12043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: acquiring CA information for follower catalog manager: success
I20260925 14:59:00.326975 12043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:59:00.576725 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:00.577020 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:00.577190 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:00.584146 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:00.584257 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:00.584415 11981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:59:00.594385 11981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab
--principal=oryx/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:39633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=37515
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:00.595635 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:00.596954 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:00.610399 12051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.610484 12048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.610460 12049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:00.611389 11981 server_base.cc:1034] running on GCE node
I20260925 14:59:00.611996 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:00.612884 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:00.614121 11981 hybrid_clock.cc:648] HybridClock initialized: now 1790348340614100 us; error 78 us; skew 500 ppm
I20260925 14:59:00.614466 11981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348340, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:00.617561 11981 init.cc:377] Logged in from keytab as oryx/127.4.243.193@KRBTEST.COM (short username oryx)
I20260925 14:59:00.619434 11981 webserver.cc:460] Webserver started at http://127.4.243.193:37515/ using document root <none> and password file <none>
I20260925 14:59:00.620072 11981 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:00.620216 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:00.625073 11981 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 14:59:00.627573 12058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.628763 11981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:00.628930 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "b08a3dae84034ac2aba9be8b32c40894"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-qxh0"
server_key: "bd4c7441c66967a35ed9da434b921cd0"
server_key_iv: "7eea4eaf8390caf912980f31296a1bb0"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.629801 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:00.647994 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:00.652807 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:00.653102 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:00.654127 11981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:00.655974 11981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:00.656082 11981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.656172 11981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:00.656230 11981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.691079 11981 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:39633
I20260925 14:59:00.691126 12170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:39633 every 8 connection(s)
I20260925 14:59:00.692435 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:00.700081 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 11981
I20260925 14:59:00.700436 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11402
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348340, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.252@KRBTEST.COM
I20260925 14:59:00.728698 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.701311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.4.243.252:35727)
0925 14:59:00.721313 (+ 20002us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.721318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.721332 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:59:00.721405 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.721409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.721491 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.721603 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:00.722222 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.723413 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.725438 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.726943 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.727098 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.727808 (+ 710us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.728217 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.728341 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19882,"thread_start_us":20445,"threads_started":1}
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348340, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.253@KRBTEST.COM
I20260925 14:59:00.728655 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.701128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.0.0.1:60762)
0925 14:59:00.715404 (+ 14276us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.715422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.715533 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.721851 (+ 6318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.721856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.721885 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:00.722092 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.722105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.723547 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.723556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.725302 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.725313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.727933 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.727938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.728062 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.728084 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.728196 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.728329 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14165,"thread_start_us":14087,"threads_started":1}
I20260925 14:59:00.730046 12004 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 14:59:00.730855 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4bf6ce139749de929c37851c302282. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:00.731141 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: This master's current role is: LEARNER
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.194@KRBTEST.COM
I20260925 14:59:00.734169 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.695125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:49043 (local address 127.4.243.252:35727)
0925 14:59:00.695625 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.695631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.707030 (+ 11399us) server_negotiation.cc:407] Connection header received
0925 14:59:00.711631 (+ 4601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.711639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.711735 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.711820 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.714503 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.716334 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.720006 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.721244 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.721349 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.726384 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.726391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.726395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.726432 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.730486 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.731506 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.731514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.731521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.731651 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.732194 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.732199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.732203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.732534 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.732692 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.733593 (+ 901us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.733792 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260925 14:59:00.736436 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.695758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.4.243.193:49043)
0925 14:59:00.706857 (+ 11099us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.706925 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.711469 (+ 4544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.712379 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.712394 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.713376 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.714275 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.714302 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.716570 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.716577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.719078 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.719096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.721713 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.721722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.722043 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.722831 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.722885 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.726122 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.730664 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.730698 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.730716 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.731385 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.731788 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.731795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.731805 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.732028 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.732671 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.732679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.733136 (+ 457us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.734450 (+ 1314us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.735044 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10747,"thread_start_us":23451,"threads_started":1}
I20260925 14:59:00.737840 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.695881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:47471 (local address 127.4.243.253:37649)
0925 14:59:00.707391 (+ 11510us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.707400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.719739 (+ 12339us) server_negotiation.cc:407] Connection header received
0925 14:59:00.719983 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.719989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.720077 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.720175 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.721667 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.723120 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.725184 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.725790 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.725916 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.729722 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.729729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.729733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.729771 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.734291 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.735209 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.735214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.735220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.735759 (+ 539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.736190 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.736198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.736203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.736580 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.736707 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.737231 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.737434 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11368,"thread_start_us":14723,"threads_started":1}
I20260925 14:59:00.737957 12171 heartbeater.cc:347] Connected to a master server at 127.4.243.252:35727
Sep 25 14:59:00 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348340, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:00.738260 12171 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:00.738914 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.695758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.4.243.193:47471)
0925 14:59:00.719612 (+ 23854us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.719637 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.719837 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.720376 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.720381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.721215 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.721508 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.721519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.723307 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.723317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.725029 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.725042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.726175 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.726180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.726398 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.726962 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.726987 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.729501 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.734439 (+ 4938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.734445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.734449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.735062 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.735910 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.735914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.735919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.736065 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.736784 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.736788 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.736932 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.738057 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.738389 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23705,"mutex_wait_us":624,"thread_start_us":11141,"threads_started":1}
I20260925 14:59:00.739207 12171 heartbeater.cc:511] Master 127.4.243.252:35727 requested a full tablet report, sending...
I20260925 14:59:00.739729 12173 heartbeater.cc:347] Connected to a master server at 127.4.243.253:37649
I20260925 14:59:00.739859 12173 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:00.740334 12173 heartbeater.cc:511] Master 127.4.243.253:37649 requested a full tablet report, sending...
I20260925 14:59:00.741634 11994 ts_manager.cc:194] Registered new tserver with Master: b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633)
I20260925 14:59:00.741952 11805 ts_manager.cc:194] Registered new tserver with Master: b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633)
I20260925 14:59:00.743904 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.702764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.4.243.194:46197)
0925 14:59:00.713031 (+ 10267us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.713042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.713078 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:59:00.713157 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.713169 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.713452 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.713651 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.717688 (+ 4037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.718692 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.720627 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.721487 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.721644 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.735049 (+ 13405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.735065 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.735082 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.735130 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.740073 (+ 4943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.740882 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.740893 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.740902 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.741069 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.741533 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.741538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.741542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.741971 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.742285 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.743384 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.743623 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10083,"thread_start_us":9938,"threads_started":1}
I20260925 14:59:00.745147 11525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:00.746949 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.696198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.193:38159)
0925 14:59:00.707905 (+ 11707us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.707925 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.711118 (+ 3193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.724523 (+ 13405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.724531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.724996 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.725233 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.725244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.730937 (+ 5693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.730942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.732548 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.732564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.735041 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.735047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.735228 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.735930 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:00.735960 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.739481 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.743697 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.743700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.743702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.743988 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.745138 (+ 1150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:00.745141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:00.745144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:00.745313 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:00.745924 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.745928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.746121 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.746248 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.746566 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11552,"thread_start_us":10197,"threads_started":1}
I20260925 14:59:00.747400 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.696617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:38159 (local address 127.4.243.254:38093)
0925 14:59:00.724211 (+ 27594us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.724218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.724239 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:59:00.724286 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.724291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.724350 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.727279 (+ 2929us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.727505 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.730768 (+ 3263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.733162 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.733799 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.735475 (+ 1676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.739698 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.739704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.739708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:00.739742 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.743575 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.744836 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.744841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.744844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.745019 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:00.745433 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:00.745438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:00.745441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:00.745752 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.745932 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.746767 (+ 835us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.746971 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26017,"thread_start_us":26859,"threads_started":1}
I20260925 14:59:00.748054 12172 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38093
I20260925 14:59:00.748175 12172 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:00.748603 12172 heartbeater.cc:511] Master 127.4.243.254:38093 requested a full tablet report, sending...
I20260925 14:59:00.749814 11207 ts_manager.cc:194] Re-registered known tserver with Master: b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633)
I20260925 14:59:00.751045 11207 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:38159
I20260925 14:59:00.832129 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11402
I20260925 14:59:00.848263 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:46197
--local_ip_for_outbound_sockets=127.4.243.194
--tserver_master_addrs=127.4.243.254:38093
--webserver_port=35245
--webserver_interface=127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab
--principal=oryx/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:01.159935 12186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:01.160243 12186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:01.160425 12186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:01.167538 12186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:01.167636 12186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:01.167809 12186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:59:01.178087 12186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab
--principal=oryx/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:46197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=35245
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:01.179440 12186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:01.180760 12186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:01.194099 12191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:01.194139 12194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:01.194188 12192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:01.196681 11955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: NON_VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: true } } }
I20260925 14:59:01.197067 11955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:01.198660 11955 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: attempting to promote NON_VOTER 4f029608a0734b91877cda219816e8f4 to VOTER
I20260925 14:59:01.199221 11955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } }
I20260925 14:59:01.200261 12186 server_base.cc:1034] running on GCE node
I20260925 14:59:01.201041 12186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:01.201357 12004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 LEARNER]: Refusing update from remote peer 6b4bf6ce139749de929c37851c302282: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 14:59:01.201373 11815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Refusing update from remote peer 6b4bf6ce139749de929c37851c302282: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 14:59:01.201897 12197 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 14:59:01.202397 12186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:01.202373 12196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 14:59:01.203634 12186 hybrid_clock.cc:648] HybridClock initialized: now 1790348341203556 us; error 65 us; skew 500 ppm
I20260925 14:59:01.203984 12186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:01.205888 11815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f029608a0734b91877cda219816e8f4 (127.4.243.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:01.205194 12197 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f029608a0734b91877cda219816e8f4 (127.4.243.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:01.206933 12004 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f029608a0734b91877cda219816e8f4 (127.4.243.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:01.208353 12186 init.cc:377] Logged in from keytab as oryx/127.4.243.194@KRBTEST.COM (short username oryx)
I20260925 14:59:01.211097 11955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:01.211174 12196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:01.211557 11955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:01.211591 12196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:01.211931 12196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:01.212225 12196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:01.212818 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:01.213146 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:01.215101 12186 webserver.cc:460] Webserver started at http://127.4.243.194:35245/ using document root <none> and password file <none>
I20260925 14:59:01.216068 12186 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:01.216215 12186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:01.217558 12204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b4bf6ce139749de929c37851c302282" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:01.217947 12204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:01.222184 12186 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:01.224784 12208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:01.226238 12186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:59:01.226395 12186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "f2e7fdf2d9774f679abcb8c82993ec2b"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1e7dde8b7c9acfc91ccc7058e2b3013"
server_key_iv: "598294ba1f3cfe569b82c1e7a9541c89"
server_key_version: "encryptionkey@0"
I20260925 14:59:01.227368 12186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:01.240577 12186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:01.244729 12186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:01.245021 12186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:01.245883 12186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:01.247248 12186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:01.247381 12186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:01.247493 12186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:01.247570 12186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:01.279721 12186 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:46197
I20260925 14:59:01.279791 12320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:46197 every 8 connection(s)
I20260925 14:59:01.281096 12186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:01.287021 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 12186
I20260925 14:59:01.287271 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11535
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.195@KRBTEST.COM
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.252@KRBTEST.COM
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.253@KRBTEST.COM
I20260925 14:59:01.326329 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.288717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.4.243.195:41717)
0925 14:59:01.292602 (+ 3885us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.292614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.297873 (+ 5259us) server_negotiation.cc:407] Connection header received
0925 14:59:01.297965 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.297977 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.298242 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.298395 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.301796 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.303657 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.311097 (+ 7440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.312143 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.312261 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.319081 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.319096 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.319113 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.319153 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.322978 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.323742 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.323751 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.323755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.323902 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.324335 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.324342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.324346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.324734 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.325036 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.325656 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.325913 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3702,"thread_start_us":3558,"threads_started":1}
I20260925 14:59:01.328404 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.305393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:33077 (local address 127.4.243.252:35727)
0925 14:59:01.306076 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.306085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.306108 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:59:01.306172 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.306179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.306269 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.306385 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.309403 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.310176 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.313573 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.314283 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.314767 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.320073 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.320079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.320084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.320122 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.323557 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.324760 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.324764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.324767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.324917 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.325395 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.325400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.325405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.325752 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.325882 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.327436 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.328020 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":333,"threads_started":1}
I20260925 14:59:01.328898 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.286898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:52205 (local address 127.4.243.254:38093)
0925 14:59:01.307722 (+ 20824us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.307727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.307746 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:59:01.307803 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.307807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.307879 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.307988 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.309592 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.310691 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.312677 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.313345 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.313465 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.322238 (+ 8773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.322245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.322249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.322291 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.325575 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.326813 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.326818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.326822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.326966 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.327409 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.327414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.327417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.327714 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.327799 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.328316 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.328566 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20701,"thread_start_us":21169,"threads_started":1}
I20260925 14:59:01.329078 11658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:01.330114 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.283926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:40037 (local address 127.4.243.253:37649)
0925 14:59:01.312230 (+ 28304us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.312237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.312253 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:59:01.312311 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.312316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.312391 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.312504 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.313465 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.314503 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.316152 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.316541 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.316661 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.322000 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.322006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.322010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.322051 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.325726 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.326683 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.326695 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.326699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.326837 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.328118 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.328123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.328126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.328455 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.328588 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.329588 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.329795 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28169,"thread_start_us":28683,"threads_started":1}
I20260925 14:59:01.330652 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.304752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.4.243.194:33077)
0925 14:59:01.305432 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.305450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.305551 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.307079 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.307094 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.308118 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:01.309226 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.309248 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.310327 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.310332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.313394 (+ 3062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.313405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.315073 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.315081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.315299 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.315897 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:01.315926 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:01.319792 (+ 3866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:01.323701 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.324033 (+ 332us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.324049 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.324624 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.325048 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.325053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.325057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.325267 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.325953 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:01.325960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:01.326374 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.326816 (+ 442us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.328781 (+ 1965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":76,"spinlock_wait_cycles":1664,"thread_start_us":761,"threads_started":1}
I20260925 14:59:01.330729 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.284595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.194:52205)
0925 14:59:01.294228 (+ 9633us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.294270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.297346 (+ 3076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.308068 (+ 10722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.308076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.308952 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:01.309401 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.309413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.310821 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.310826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.312506 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.312517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.313737 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.313744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.314064 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.315739 (+ 1675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:01.317362 (+ 1623us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:01.321709 (+ 4347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:01.325712 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.325716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.325720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.326700 (+ 980us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.327104 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.327109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.327113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.327263 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.327869 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:01.327872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:01.328054 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.329078 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.329393 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9288,"mutex_wait_us":55,"spinlock_wait_cycles":18304,"thread_start_us":9301,"threads_started":1}
I20260925 14:59:01.330871 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.285294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.4.243.194:40037)
0925 14:59:01.304319 (+ 19025us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.304342 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.304443 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.312518 (+ 8075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.312523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.313040 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:01.313290 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.313301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.314668 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.314673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.315973 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.316030 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.316724 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.316732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.316929 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.317513 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:01.317529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:01.321756 (+ 4227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:01.325876 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.325882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.325886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.326413 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.326978 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.326982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.326986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.327979 (+ 993us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.328659 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:01.328664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:01.328801 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.328959 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.329206 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18899,"spinlock_wait_cycles":34048,"thread_start_us":18748,"threads_started":1}
I20260925 14:59:01.332113 12323 heartbeater.cc:347] Connected to a master server at 127.4.243.253:37649
I20260925 14:59:01.332417 12323 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:01.333149 12323 heartbeater.cc:511] Master 127.4.243.253:37649 requested a full tablet report, sending...
I20260925 14:59:01.333786 12322 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38093
I20260925 14:59:01.333906 12322 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:01.334280 12322 heartbeater.cc:511] Master 127.4.243.254:38093 requested a full tablet report, sending...
I20260925 14:59:01.334865 11805 ts_manager.cc:194] Registered new tserver with Master: f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197)
I20260925 14:59:01.335379 11207 ts_manager.cc:194] Re-registered known tserver with Master: f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197)
I20260925 14:59:01.336409 12321 heartbeater.cc:347] Connected to a master server at 127.4.243.252:35727
I20260925 14:59:01.336508 12321 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:01.336786 12321 heartbeater.cc:511] Master 127.4.243.252:35727 requested a full tablet report, sending...
I20260925 14:59:01.336790 11207 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:52205
I20260925 14:59:01.337756 11994 ts_manager.cc:194] Registered new tserver with Master: f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197)
I20260925 14:59:01.419418 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11535
I20260925 14:59:01.436429 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:41717
--local_ip_for_outbound_sockets=127.4.243.195
--tserver_master_addrs=127.4.243.254:38093
--webserver_port=42975
--webserver_interface=127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab
--principal=oryx/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:01.751812 12334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:01.752126 12334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:01.752297 12334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 14:59:01.753710 12172 heartbeater.cc:503] Master 127.4.243.254:38093 was elected leader, sending a full tablet report...
W20260925 14:59:01.759758 12334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:01.759850 12334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:01.760083 12334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:59:01.769295 12334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab
--principal=oryx/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:41717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=42975
--tserver_master_addrs=127.4.243.254:38093,127.4.243.253:37649,127.4.243.252:35727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:01.770447 12334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:01.771800 12334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:01.784176 12339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:01.784397 12340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:01.785363 12334 server_base.cc:1034] running on GCE node
W20260925 14:59:01.786010 12342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:01.786664 12334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:01.787539 12334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:01.788753 12334 hybrid_clock.cc:648] HybridClock initialized: now 1790348341788712 us; error 25 us; skew 500 ppm
I20260925 14:59:01.789086 12334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:01.792135 12334 init.cc:377] Logged in from keytab as oryx/127.4.243.195@KRBTEST.COM (short username oryx)
I20260925 14:59:01.793442 12334 webserver.cc:460] Webserver started at http://127.4.243.195:42975/ using document root <none> and password file <none>
I20260925 14:59:01.794095 12334 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:01.794199 12334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:01.799175 12334 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:59:01.802718 12350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:01.804203 12334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:59:01.804332 12334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "db0f5351dab84cf89d80a37e0b7be0ed"
format_stamp: "Formatted at 2026-09-25 14:58:58 on dist-test-slave-qxh0"
server_key: "f1f7b01de59481a58a2f19ace389cc03"
server_key_iv: "69cd48f2444825234ad8124f13b5671f"
server_key_version: "encryptionkey@0"
I20260925 14:59:01.805258 12334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:01.830508 12334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:01.834560 12334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:01.834867 12334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:01.835808 12334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:01.837126 12334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:01.837205 12334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:01.837280 12334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:01.837333 12334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:01.869211 12334 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:41717
I20260925 14:59:01.869329 12462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:41717 every 8 connection(s)
I20260925 14:59:01.870581 12334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:01.875849 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 12334
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.253@KRBTEST.COM
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.252@KRBTEST.COM
Sep 25 14:59:01 dist-test-slave-qxh0 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:01.903807 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.873427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:36293 (local address 127.4.243.253:37649)
0925 14:59:01.879787 (+ 6360us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.879796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.879817 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:59:01.879885 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.879892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.879982 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.880106 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.882706 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.883736 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.887599 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.888295 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.888422 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.896097 (+ 7675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.896104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.896109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.896160 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.900122 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.901166 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.901174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.901177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.901325 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.901798 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.901803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.901806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.902182 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.902331 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.903216 (+ 885us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.903415 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6216,"thread_start_us":6114,"threads_started":1}
I20260925 14:59:01.905345 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.874071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.4.243.195:36293)
0925 14:59:01.878935 (+ 4864us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.878956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.879075 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.880408 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.880419 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.881195 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:01.882262 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.882287 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.883891 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.883896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.887452 (+ 3556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.887471 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.888493 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.888500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.888830 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.890188 (+ 1358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:01.890226 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:01.895887 (+ 5661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:01.900312 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.900323 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.900356 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.901050 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.901422 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.901428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.901434 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.901697 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.902305 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:01.902313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:01.902792 (+ 479us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.903118 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.903595 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4697,"mutex_wait_us":447,"spinlock_wait_cycles":16256,"thread_start_us":6684,"threads_started":1}
I20260925 14:59:01.910401 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.874071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.4.243.195:51677)
0925 14:59:01.874881 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.874929 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.877453 (+ 2524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.880959 (+ 3506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.880965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.881352 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:01.882262 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.882287 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.883824 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.883830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.885662 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.885676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.889483 (+ 3807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.889496 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.889761 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.892825 (+ 3064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:01.892852 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:01.896655 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:01.907192 (+ 10537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.907197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.907201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.907701 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.908135 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.908139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.908143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.908302 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.909073 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:01.909080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:01.909258 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.909402 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.909818 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":43,"thread_start_us":512,"threads_started":1}
I20260925 14:59:01.911729 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.874616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:51677 (local address 127.4.243.252:35727)
0925 14:59:01.880611 (+ 5995us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.880618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.880638 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:59:01.880701 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.880708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.880790 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.880916 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.882466 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.883634 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.887714 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.888325 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.896838 (+ 8513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.897074 (+ 236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.897083 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.897103 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.897143 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.907049 (+ 9906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.907865 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.907870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.907874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.908006 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.908517 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.908525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.908530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.908931 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.910822 (+ 1891us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.910976 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.911248 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5863,"thread_start_us":5758,"threads_started":1}
I20260925 14:59:01.912946 12463 heartbeater.cc:347] Connected to a master server at 127.4.243.252:35727
I20260925 14:59:01.913472 12463 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:01.914744 12463 heartbeater.cc:511] Master 127.4.243.252:35727 requested a full tablet report, sending...
I20260925 14:59:01.916898 11994 ts_manager.cc:194] Registered new tserver with Master: db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:59:01.919485 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.877348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.195:39417)
0925 14:59:01.881784 (+ 4436us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.881807 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.882091 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.882781 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.882788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.883595 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:01.884016 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.884036 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.885512 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.885519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.895386 (+ 9867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.895407 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.896966 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.896974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.897211 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.898229 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:01.898256 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:01.902028 (+ 3772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:01.911361 (+ 9333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.911368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.911374 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.912182 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.912636 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:01.912640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:01.912644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:01.912791 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:01.918514 (+ 5723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:01.918522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:01.918744 (+ 222us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.918889 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.919135 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3972,"thread_start_us":1328,"threads_started":1}
I20260925 14:59:01.927186 12465 heartbeater.cc:347] Connected to a master server at 127.4.243.253:37649
I20260925 14:59:01.927398 12465 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:01.928596 12465 heartbeater.cc:511] Master 127.4.243.253:37649 requested a full tablet report, sending...
I20260925 14:59:01.928694 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.874179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:39417 (local address 127.4.243.254:38093)
0925 14:59:01.878230 (+ 4051us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.878239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.881891 (+ 3652us) server_negotiation.cc:407] Connection header received
0925 14:59:01.882306 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.882313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.882417 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.882510 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.884233 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.885310 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.895761 (+ 10451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.896544 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.896653 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.905968 (+ 9315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.905990 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.905997 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.906296 (+ 299us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.911148 (+ 4852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.912312 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.912331 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.912336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.912506 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.918006 (+ 5500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.918014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.918019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.918373 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.918521 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.927578 (+ 9057us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.927864 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3903,"thread_start_us":3748,"threads_started":1}
I20260925 14:59:01.929699 12464 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38093
I20260925 14:59:01.929919 12464 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:01.930475 12464 heartbeater.cc:511] Master 127.4.243.254:38093 requested a full tablet report, sending...
I20260925 14:59:01.931694 11805 ts_manager.cc:194] Registered new tserver with Master: db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:59:01.931948 11207 ts_manager.cc:194] Re-registered known tserver with Master: db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:59:01.933219 11207 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:39417
I20260925 14:59:01.935164 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.885814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.4.243.253:37649)
0925 14:59:01.904400 (+ 18586us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.904405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.904423 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:59:01.904467 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.904471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.904539 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.904630 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.912473 (+ 7843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.913566 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.916125 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.916779 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.916886 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.920256 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.920262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.920268 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.920308 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.928536 (+ 8228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.930183 (+ 1647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.930189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.930192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.930354 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.930843 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.930847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.930850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.931198 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.931426 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.934523 (+ 3097us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.934787 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18108,"thread_start_us":34747,"threads_started":1}
I20260925 14:59:01.942893 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.885820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.4.243.254:38093)
0925 14:59:01.912544 (+ 26724us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.912552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.912568 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:59:01.912611 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.912616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.912682 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.912772 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.929021 (+ 16249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.930268 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.932511 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.933434 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.933560 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.937169 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.937175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.937181 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.937221 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.940321 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.941040 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.941045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.941048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.941158 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.941530 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.941534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.941537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.941766 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.941959 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.942284 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.942490 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":26610,"thread_start_us":26469,"threads_started":1}
I20260925 14:59:01.943921 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.892627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.4.243.252:35727)
0925 14:59:01.902974 (+ 10347us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.902982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.913498 (+ 10516us) server_negotiation.cc:407] Connection header received
0925 14:59:01.913548 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.913552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.913619 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.913702 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:01.916630 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.918115 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.922675 (+ 4560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.923259 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.923369 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.938239 (+ 14870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:01.938244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:01.938247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:01.938279 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:01.941384 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.941958 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.941964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.941967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.942082 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:01.942446 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:01.942450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:01.942454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:01.942790 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:01.943026 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.943328 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.943542 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10199,"thread_start_us":10087,"threads_started":1}
I20260925 14:59:01.946534 11207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:01.953078 11207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:01.981902 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.972826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42562)
0925 14:59:01.973745 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.973772 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.973913 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.974901 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.974905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.974928 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:01.975197 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.975205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.976983 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.976988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.979038 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.979052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.980832 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.980844 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.981019 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.981053 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.981118 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.981243 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":683,"threads_started":1}
I20260925 14:59:01.982885 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.973362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.4.243.193:39633)
0925 14:59:01.974363 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.974371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.974393 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:59:01.974488 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.974497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.974737 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.974913 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:01.975352 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.976720 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.979239 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.980616 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.982154 (+ 1538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.982353 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.982432 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.982564 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":734,"threads_started":1}
I20260925 14:59:01.983738 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.971060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41717 (local address 127.0.0.1:48056)
0925 14:59:01.971605 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.971626 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.971751 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.974355 (+ 2604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.974359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.974383 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:01.974686 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.974697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.977518 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.977523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.980653 (+ 3130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.980672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.982712 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.982718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.982864 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.982900 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.983195 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.983369 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
I20260925 14:59:01.983752 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.971352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.4.243.195:41717)
0925 14:59:01.972581 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.972595 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.972643 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:59:01.972787 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.972800 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.973163 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.973680 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:01.974869 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.977282 (+ 2413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.981024 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.982503 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.982673 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.982909 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.983261 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.983425 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":372,"threads_started":1}
I20260925 14:59:01.985423 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.970690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.4.243.194:46197)
0925 14:59:01.977783 (+ 7093us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:01.977789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:01.977818 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:59:01.977888 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:01.977893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:01.978117 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:01.978313 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:01.978959 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.979909 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.982849 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.984281 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.984428 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.984626 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:01.984973 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 14:59:01.985140 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6929,"thread_start_us":6830,"threads_started":1}
I20260925 14:59:01.985491 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:01.970339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:46197 (local address 127.0.0.1:43630)
0925 14:59:01.970914 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:01.970934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:01.971061 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:01.978297 (+ 7236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:01.978302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:01.978339 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:01.978782 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.978792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.980046 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.980054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.982510 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:01.982522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:01.984490 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:01.984496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:01.984691 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:01.984744 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 14:59:01.984899 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 14:59:01.985041 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":1,"thread_start_us":311,"threads_started":1}
I20260925 14:59:01.986477 12106 tablet_service.cc:1467] Processing CreateTablet for tablet c76b5d1c3cb64e8498cfce0c28479eeb (DEFAULT_TABLE table=test-table [id=d2cae10f91904a95871696486088b812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:01.987964 12106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76b5d1c3cb64e8498cfce0c28479eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:01.989099 12256 tablet_service.cc:1467] Processing CreateTablet for tablet c76b5d1c3cb64e8498cfce0c28479eeb (DEFAULT_TABLE table=test-table [id=d2cae10f91904a95871696486088b812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:01.989215 12398 tablet_service.cc:1467] Processing CreateTablet for tablet c76b5d1c3cb64e8498cfce0c28479eeb (DEFAULT_TABLE table=test-table [id=d2cae10f91904a95871696486088b812]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:01.990334 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76b5d1c3cb64e8498cfce0c28479eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:01.990368 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76b5d1c3cb64e8498cfce0c28479eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:02.002835 12497 tablet_bootstrap.cc:492] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Bootstrap starting.
I20260925 14:59:02.005632 12499 tablet_bootstrap.cc:492] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Bootstrap starting.
I20260925 14:59:02.008240 12497 tablet_bootstrap.cc:654] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:02.009167 12499 tablet_bootstrap.cc:654] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:02.009730 12497 log.cc:824] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:02.010389 12499 log.cc:824] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:02.010913 12498 tablet_bootstrap.cc:492] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Bootstrap starting.
I20260925 14:59:02.013393 12499 tablet_bootstrap.cc:492] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: No bootstrap required, opened a new log
I20260925 14:59:02.013643 12499 ts_tablet_manager.cc:1397] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 14:59:02.013689 12497 tablet_bootstrap.cc:492] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: No bootstrap required, opened a new log
I20260925 14:59:02.013969 12497 ts_tablet_manager.cc:1397] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260925 14:59:02.021579 12499 raft_consensus.cc:348] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.021680 12497 raft_consensus.cc:348] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.021915 12499 raft_consensus.cc:374] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:02.022011 12499 raft_consensus.cc:729] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e7fdf2d9774f679abcb8c82993ec2b, State: Initialized, Role: FOLLOWER
I20260925 14:59:02.022035 12497 raft_consensus.cc:374] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:02.022126 12497 raft_consensus.cc:729] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0f5351dab84cf89d80a37e0b7be0ed, State: Initialized, Role: FOLLOWER
I20260925 14:59:02.022576 12499 consensus_queue.cc:260] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.022737 12497 consensus_queue.cc:260] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.024209 12499 ts_tablet_manager.cc:1428] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
I20260925 14:59:02.024446 12322 heartbeater.cc:503] Master 127.4.243.254:38093 was elected leader, sending a full tablet report...
I20260925 14:59:02.025138 12498 tablet_bootstrap.cc:654] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:02.026731 12498 log.cc:824] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:02.031100 12497 ts_tablet_manager.cc:1428] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260925 14:59:02.031417 12464 heartbeater.cc:503] Master 127.4.243.254:38093 was elected leader, sending a full tablet report...
I20260925 14:59:02.035624 12498 tablet_bootstrap.cc:492] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: No bootstrap required, opened a new log
I20260925 14:59:02.035895 12498 ts_tablet_manager.cc:1397] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.003s
I20260925 14:59:02.043376 12498 raft_consensus.cc:348] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.043699 12498 raft_consensus.cc:374] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:02.043850 12498 raft_consensus.cc:729] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08a3dae84034ac2aba9be8b32c40894, State: Initialized, Role: FOLLOWER
I20260925 14:59:02.044386 12498 consensus_queue.cc:260] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.046069 12498 ts_tablet_manager.cc:1428] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
W20260925 14:59:02.059546 12324 tablet.cc:2369] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:02.123518 12466 tablet.cc:2369] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:02.196445 12174 tablet.cc:2369] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:59:02.301465 12505 raft_consensus.cc:482] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:02.301797 12505 raft_consensus.cc:504] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.303474 12505 leader_election.cc:288] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197), db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:59:02.309908 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.303844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:38239 (local address 127.4.243.194:46197)
0925 14:59:02.304014 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.304019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.304468 (+ 449us) server_negotiation.cc:407] Connection header received
0925 14:59:02.304782 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.304791 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.304895 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.304960 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:02.305594 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.306279 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.308180 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.308965 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.309148 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.309217 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.309543 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.309662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:59:02.310132 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.303738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:46197 (local address 127.4.243.193:38239)
0925 14:59:02.304374 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.304392 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.304579 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.305129 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.305132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.305176 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:02.305446 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.305453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.306411 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.306416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.308013 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.308025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.309166 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.309172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.309388 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.309411 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.309520 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.309637 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":59,"spinlock_wait_cycles":17152,"thread_start_us":323,"threads_started":1}
I20260925 14:59:02.310227 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.304552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:39861 (local address 127.4.243.195:41717)
0925 14:59:02.304729 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.304737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.305112 (+ 375us) server_negotiation.cc:407] Connection header received
0925 14:59:02.305338 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.305342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.305399 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.305491 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:02.305949 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.306670 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.308369 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.309296 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.309470 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.309540 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.309878 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.309992 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:59:02.310436 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.304452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41717 (local address 127.4.243.193:39861)
0925 14:59:02.305008 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.305021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.305182 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.305504 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.305509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.305530 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:02.305782 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.305790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.306794 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.306799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.308223 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.308234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.309501 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.309507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.309652 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.309673 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.309854 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.309971 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":3,"spinlock_wait_cycles":3840,"thread_start_us":325,"threads_started":1}
I20260925 14:59:02.311193 12276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76b5d1c3cb64e8498cfce0c28479eeb" candidate_uuid: "b08a3dae84034ac2aba9be8b32c40894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" is_pre_election: true
I20260925 14:59:02.311345 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76b5d1c3cb64e8498cfce0c28479eeb" candidate_uuid: "b08a3dae84034ac2aba9be8b32c40894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" is_pre_election: true
I20260925 14:59:02.311630 12276 raft_consensus.cc:2454] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08a3dae84034ac2aba9be8b32c40894 in term 0.
I20260925 14:59:02.311821 12418 raft_consensus.cc:2454] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08a3dae84034ac2aba9be8b32c40894 in term 0.
I20260925 14:59:02.312222 12061 leader_election.cc:302] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b08a3dae84034ac2aba9be8b32c40894, f2e7fdf2d9774f679abcb8c82993ec2b; no voters:
I20260925 14:59:02.312541 12505 raft_consensus.cc:2781] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:02.312683 12505 raft_consensus.cc:482] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:02.312829 12505 raft_consensus.cc:3037] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:02.315064 12505 raft_consensus.cc:504] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.315608 12505 leader_election.cc:288] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [CANDIDATE]: Term 1 election: Requested vote from peers f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197), db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:59:02.316172 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76b5d1c3cb64e8498cfce0c28479eeb" candidate_uuid: "b08a3dae84034ac2aba9be8b32c40894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0f5351dab84cf89d80a37e0b7be0ed"
I20260925 14:59:02.316372 12418 raft_consensus.cc:3037] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:02.316625 12276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76b5d1c3cb64e8498cfce0c28479eeb" candidate_uuid: "b08a3dae84034ac2aba9be8b32c40894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b"
I20260925 14:59:02.316839 12276 raft_consensus.cc:3037] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:02.318665 12418 raft_consensus.cc:2454] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08a3dae84034ac2aba9be8b32c40894 in term 1.
I20260925 14:59:02.319031 12276 raft_consensus.cc:2454] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08a3dae84034ac2aba9be8b32c40894 in term 1.
I20260925 14:59:02.319078 12061 leader_election.cc:302] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b08a3dae84034ac2aba9be8b32c40894, db0f5351dab84cf89d80a37e0b7be0ed; no voters:
I20260925 14:59:02.319505 12505 raft_consensus.cc:2781] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:02.319895 12505 raft_consensus.cc:686] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 1 LEADER]: Becoming Leader. State: Replica: b08a3dae84034ac2aba9be8b32c40894, State: Running, Role: LEADER
I20260925 14:59:02.320415 12505 consensus_queue.cc:237] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } }
I20260925 14:59:02.325136 11206 catalog_manager.cc:5495] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 reported cstate change: term changed from 0 to 1, leader changed from <none> to b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "b08a3dae84034ac2aba9be8b32c40894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } health_report { overall_health: HEALTHY } } }
I20260925 14:59:02.344641 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.340041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.4.243.193:39633)
0925 14:59:02.340198 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.340203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.340217 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:59:02.340283 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.340286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.340355 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.340474 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:02.340806 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.341509 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.342849 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.343204 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.343297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.343648 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:02.344108 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:02.344210 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.344289 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.344355 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:59:02.351964 12418 raft_consensus.cc:1261] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Refusing update from remote peer b08a3dae84034ac2aba9be8b32c40894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:02.352031 12276 raft_consensus.cc:1261] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 FOLLOWER]: Refusing update from remote peer b08a3dae84034ac2aba9be8b32c40894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:02.352813 12508 consensus_queue.cc:1035] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:02.353159 12505 consensus_queue.cc:1035] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:02.385963 11206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:02.386214 11206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:02.392470 11206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b4bf6ce139749de929c37851c302282: Sending DeleteTablet for 3 replicas of tablet c76b5d1c3cb64e8498cfce0c28479eeb
I20260925 14:59:02.393977 12256 tablet_service.cc:1514] Processing DeleteTablet for tablet c76b5d1c3cb64e8498cfce0c28479eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:02 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:43630
I20260925 14:59:02.394179 12398 tablet_service.cc:1514] Processing DeleteTablet for tablet c76b5d1c3cb64e8498cfce0c28479eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:02 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:48056
I20260925 14:59:02.394531 12106 tablet_service.cc:1514] Processing DeleteTablet for tablet c76b5d1c3cb64e8498cfce0c28479eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:02 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:42562
I20260925 14:59:02.405130 12521 tablet_replica.cc:330] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: stopping tablet replica
I20260925 14:59:02.405776 12521 raft_consensus.cc:2229] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:02.406399 12521 raft_consensus.cc:2258] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:02.408911 12521 ts_tablet_manager.cc:1905] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:02.417927 12521 ts_tablet_manager.cc:1918] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:02.418167 12521 log.cc:1196] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/c76b5d1c3cb64e8498cfce0c28479eeb
I20260925 14:59:02.418946 12521 ts_tablet_manager.cc:1939] T c76b5d1c3cb64e8498cfce0c28479eeb P db0f5351dab84cf89d80a37e0b7be0ed: Deleting consensus metadata
I20260925 14:59:02.421000 11193 catalog_manager.cc:4841] TS db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717): tablet c76b5d1c3cb64e8498cfce0c28479eeb (table test-table [id=d2cae10f91904a95871696486088b812]) successfully deleted
I20260925 14:59:02.423478 12522 tablet_replica.cc:330] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: stopping tablet replica
I20260925 14:59:02.423601 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11175
I20260925 14:59:02.423934 12522 raft_consensus.cc:2229] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:02.424590 12522 raft_consensus.cc:2258] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:02.425631 11237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:02.425865 12520 tablet_replica.cc:330] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: stopping tablet replica
I20260925 14:59:02.426222 12520 raft_consensus.cc:2229] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:02.426872 12520 raft_consensus.cc:2258] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:02.428839 12522 ts_tablet_manager.cc:1905] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:02.431689 12520 ts_tablet_manager.cc:1905] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:02.439851 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.397210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.4.243.253:37649)
0925 14:59:02.397437 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.397446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.422035 (+ 24589us) server_negotiation.cc:407] Connection header received
0925 14:59:02.422250 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.422256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.422331 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.422422 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.425316 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.426392 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.428698 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.429354 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.429459 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.433278 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.433285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.433290 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.433325 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.436773 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.437527 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.437533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.437537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.437662 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.438089 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.438095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.438102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.438452 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.438751 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.439285 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.439486 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:59:02.445604 12522 ts_tablet_manager.cc:1918] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:02.445803 12522 log.cc:1196] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/c76b5d1c3cb64e8498cfce0c28479eeb
I20260925 14:59:02.446556 12522 ts_tablet_manager.cc:1939] T c76b5d1c3cb64e8498cfce0c28479eeb P b08a3dae84034ac2aba9be8b32c40894: Deleting consensus metadata
I20260925 14:59:02.446810 12520 ts_tablet_manager.cc:1918] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:02.446969 12520 log.cc:1196] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/c76b5d1c3cb64e8498cfce0c28479eeb
I20260925 14:59:02.447650 12520 ts_tablet_manager.cc:1939] T c76b5d1c3cb64e8498cfce0c28479eeb P f2e7fdf2d9774f679abcb8c82993ec2b: Deleting consensus metadata
I20260925 14:59:02.455978 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.422797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.4.243.252:35727)
0925 14:59:02.422962 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.422967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.440762 (+ 17795us) server_negotiation.cc:407] Connection header received
0925 14:59:02.441015 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.441020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.441106 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.441177 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.443021 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.444085 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.446514 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.447237 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.447348 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.450622 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.450631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.450639 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.450675 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.453323 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.453957 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.453963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.453968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.454095 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.454453 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.454460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.454463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.454810 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.455063 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.455386 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.455608 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:59:02.573130 11193 catalog_manager.cc:4841] TS f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197): tablet c76b5d1c3cb64e8498cfce0c28479eeb (table test-table [id=d2cae10f91904a95871696486088b812]) successfully deleted
I20260925 14:59:02.574493 11194 catalog_manager.cc:4841] TS b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633): tablet c76b5d1c3cb64e8498cfce0c28479eeb (table test-table [id=d2cae10f91904a95871696486088b812]) successfully deleted
I20260925 14:59:02.575207 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11175
I20260925 14:59:02.619931 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.600627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.4.243.252:35727)
0925 14:59:02.600809 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.600818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.603205 (+ 2387us) server_negotiation.cc:407] Connection header received
0925 14:59:02.603462 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.603467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.603542 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.603611 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.606089 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.607189 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.608798 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.609201 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.609341 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.612953 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.612959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.612964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.613001 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.616645 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.617439 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.617446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.617450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.617611 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.618212 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.618218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.618222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.618556 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.618848 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.619143 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.619524 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:59:02.620090 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.600518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.4.243.253:37649)
0925 14:59:02.600706 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.600713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.603187 (+ 2474us) server_negotiation.cc:407] Connection header received
0925 14:59:02.603461 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.603467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.603542 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.603611 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.606699 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.607508 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.609509 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.610151 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.610277 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.613433 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.613438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.613449 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.613478 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.616859 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.617724 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.617729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.617732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.617854 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.618290 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.618298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.618303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.618651 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.618929 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.619475 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.619693 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:59:03.374840 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.373789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.193:47121)
0925 14:59:03.374411 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.374562 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":370,"threads_started":1}
W20260925 14:59:03.375275 12172 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:38093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.378038 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.377359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.194:55675)
0925 14:59:03.377817 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.377911 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
W20260925 14:59:03.378521 12322 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:38093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.380676 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.379923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.195:54059)
0925 14:59:03.380446 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.380543 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":375,"threads_started":1}
W20260925 14:59:03.380972 12464 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:38093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.894187 12535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b4bf6ce139749de929c37851c302282)
I20260925 14:59:03.894428 12535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } }
I20260925 14:59:03.895716 12535 leader_election.cc:288] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b4bf6ce139749de929c37851c302282 (127.4.243.254:38093), 4f029608a0734b91877cda219816e8f4 (127.4.243.252:35727)
I20260925 14:59:03.896781 12538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6b4bf6ce139749de929c37851c302282)
I20260925 14:59:03.897271 12538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } }
I20260925 14:59:03.897554 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.895929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37536)
0925 14:59:03.896535 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.896867 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":311,"threads_started":1}
W20260925 14:59:03.897799 11792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.899377 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.899038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37542)
0925 14:59:03.899181 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.899245 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 14:59:03.899739 11792 leader_election.cc:334] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6b4bf6ce139749de929c37851c302282 (127.4.243.254:38093): Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.905447 12538 leader_election.cc:288] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6b4bf6ce139749de929c37851c302282 (127.4.243.254:38093), 2492809c32e7400588ed011c799620e5 (127.4.243.253:37649)
I20260925 14:59:03.906060 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.895929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:35727 (local address 127.0.0.1:60808)
0925 14:59:03.897092 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.897111 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.897239 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.897775 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.897779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.897823 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.898157 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.898174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.899800 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.899812 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.902638 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.902657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.904995 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.905004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.905271 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.905336 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.905430 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.905567 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":22,"thread_start_us":799,"threads_started":1}
I20260925 14:59:03.907121 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.896210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.4.243.252:35727)
0925 14:59:03.897422 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.897428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.897443 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:59:03.897510 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.897517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.897645 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.897752 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.898310 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.899634 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.902865 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.904574 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.904724 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.904894 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.906416 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.906651 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":817,"threads_started":1}
I20260925 14:59:03.908596 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.899540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37554)
0925 14:59:03.907713 (+ 8173us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.908066 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8064,"thread_start_us":8346,"threads_started":1}
I20260925 14:59:03.908656 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2492809c32e7400588ed011c799620e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4f029608a0734b91877cda219816e8f4" is_pre_election: true
W20260925 14:59:03.908869 11980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.909029 12004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2492809c32e7400588ed011c799620e5 in term 1.
I20260925 14:59:03.909895 11791 leader_election.cc:302] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2492809c32e7400588ed011c799620e5, 4f029608a0734b91877cda219816e8f4; no voters: 6b4bf6ce139749de929c37851c302282
I20260925 14:59:03.910331 12535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:59:03.910447 12535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6b4bf6ce139749de929c37851c302282)
I20260925 14:59:03.910549 12535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:59:03.910645 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.910196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37558)
0925 14:59:03.910399 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.910489 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 14:59:03.910825 11980 leader_election.cc:334] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6b4bf6ce139749de929c37851c302282 (127.4.243.254:38093): Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.912652 12535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } }
I20260925 14:59:03.913365 12535 leader_election.cc:288] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [CANDIDATE]: Term 2 election: Requested vote from peers 6b4bf6ce139749de929c37851c302282 (127.4.243.254:38093), 4f029608a0734b91877cda219816e8f4 (127.4.243.252:35727)
I20260925 14:59:03.913870 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.913420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37570)
0925 14:59:03.913629 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.913725 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 14:59:03.913836 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2492809c32e7400588ed011c799620e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4f029608a0734b91877cda219816e8f4"
I20260925 14:59:03.914074 12004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:59:03.914568 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.899662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.4.243.253:37649)
0925 14:59:03.908376 (+ 8714us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.908381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.908665 (+ 284us) server_negotiation.cc:407] Connection header received
0925 14:59:03.908867 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.908874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.908939 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.909028 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.909812 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.910924 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.912403 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.913381 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.913507 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.913627 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.914132 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.914265 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8573,"thread_start_us":8402,"threads_started":1}
I20260925 14:59:03.914994 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.914613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37584)
0925 14:59:03.914801 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.914899 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260925 14:59:03.915210 11792 leader_election.cc:334] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6b4bf6ce139749de929c37851c302282 (127.4.243.254:38093): Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
I20260925 14:59:03.915297 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.899606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:37649 (local address 127.0.0.1:54644)
0925 14:59:03.908547 (+ 8941us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.908567 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.908703 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.909262 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.909265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.909285 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.909617 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.909628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.911057 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.911062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.912241 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.912251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.913739 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.913744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.913862 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.913883 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.914832 (+ 949us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.914975 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8748,"thread_start_us":7894,"threads_started":1}
I20260925 14:59:03.915915 11815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f029608a0734b91877cda219816e8f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2492809c32e7400588ed011c799620e5" is_pre_election: true
I20260925 14:59:03.916262 11815 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f029608a0734b91877cda219816e8f4 in current term 2: Already voted for candidate 2492809c32e7400588ed011c799620e5 in this term.
I20260925 14:59:03.916792 11978 leader_election.cc:302] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f029608a0734b91877cda219816e8f4; no voters: 2492809c32e7400588ed011c799620e5, 6b4bf6ce139749de929c37851c302282
I20260925 14:59:03.916954 12004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2492809c32e7400588ed011c799620e5 in term 2.
I20260925 14:59:03.917229 12538 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:59:03.917351 11791 leader_election.cc:302] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2492809c32e7400588ed011c799620e5, 4f029608a0734b91877cda219816e8f4; no voters: 6b4bf6ce139749de929c37851c302282
I20260925 14:59:03.917634 12535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:59:03.918049 12535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [term 2 LEADER]: Becoming Leader. State: Replica: 2492809c32e7400588ed011c799620e5, State: Running, Role: LEADER
I20260925 14:59:03.918488 12535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } }
I20260925 14:59:03.922981 12545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2492809c32e7400588ed011c799620e5. Latest consensus state: current_term: 2 leader_uuid: "2492809c32e7400588ed011c799620e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:03.923305 12545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:03.923996 12547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:03.927635 12547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:03.928385 12547 catalog_manager.cc:1259] Loaded cluster ID: f8d2a6c02ade41339d86bfab51123204
I20260925 14:59:03.928453 12547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:03.929467 12547 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:03.930173 12547 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Loaded TSK: 0
I20260925 14:59:03.931187 12547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:03.947476 11805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:03.954396 11805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:03.958997 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.958433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37598)
0925 14:59:03.958682 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.958779 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:59:03.959014 12004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [term 2 FOLLOWER]: Refusing update from remote peer 2492809c32e7400588ed011c799620e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 14:59:03.959834 12545 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 14:59:03.960235 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.959753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37614)
0925 14:59:03.959973 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.960044 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 14:59:03.960426 11792 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 -> Peer 6b4bf6ce139749de929c37851c302282 (127.4.243.254:38093): Couldn't send request to peer 6b4bf6ce139749de929c37851c302282. Status: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:59:03.962838 12538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2492809c32e7400588ed011c799620e5. Latest consensus state: current_term: 2 leader_uuid: "2492809c32e7400588ed011c799620e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:03.963164 12538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:03.964689 12545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2492809c32e7400588ed011c799620e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:03.964983 12545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:03.965389 12538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2492809c32e7400588ed011c799620e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6b4bf6ce139749de929c37851c302282" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38093 } } peers { permanent_uuid: "2492809c32e7400588ed011c799620e5" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 37649 } attrs { promote: false } } peers { permanent_uuid: "4f029608a0734b91877cda219816e8f4" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 35727 } attrs { promote: false } } }
I20260925 14:59:03.965648 12538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f029608a0734b91877cda219816e8f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:03.983546 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.973713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41717 (local address 127.0.0.1:48062)
0925 14:59:03.973841 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.973860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.973975 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.975012 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.975018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.975050 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.975414 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.975427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.977298 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.977307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.980227 (+ 2920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.980246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.982488 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.982498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.982767 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.982824 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.982908 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.983121 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 14:59:03.985083 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.974035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.4.243.195:41717)
0925 14:59:03.974617 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.974627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.974668 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:59:03.974765 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.974773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.974855 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.975601 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.975828 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.977106 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.980482 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.982182 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.984200 (+ 2018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.984349 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.984457 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.984652 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":805,"threads_started":1}
I20260925 14:59:03.992436 12398 tablet_service.cc:1467] Processing CreateTablet for tablet 1547819728da405fab49733d060531dd (DEFAULT_TABLE table=test-table [id=512c10dd926c4f3baa88636097cbcf40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:03.992955 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1547819728da405fab49733d060531dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:03.996367 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.975775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:46197 (local address 127.0.0.1:43636)
0925 14:59:03.983830 (+ 8055us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.983853 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.983982 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.991007 (+ 7025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.991011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.991058 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.991212 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.991219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.992586 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.992592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.994375 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.994387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.995642 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.995647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.995832 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.995865 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.995970 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.996098 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7925,"thread_start_us":15548,"threads_started":1}
I20260925 14:59:03.996431 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.976297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.4.243.194:46197)
0925 14:59:03.990633 (+ 14336us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.990639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.990658 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:59:03.990725 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.990730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.990828 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.991550 (+ 722us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.991661 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.992455 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.994557 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.995483 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.995633 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.995733 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.995995 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.996119 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14192,"thread_start_us":14106,"threads_started":1}
I20260925 14:59:03.997115 12256 tablet_service.cc:1467] Processing CreateTablet for tablet 1547819728da405fab49733d060531dd (DEFAULT_TABLE table=test-table [id=512c10dd926c4f3baa88636097cbcf40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:03.997495 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1547819728da405fab49733d060531dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:03.999159 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.973422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42578)
0925 14:59:03.973725 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.973750 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.973880 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.993248 (+ 19368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.993252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.993277 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.993551 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.993563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.994883 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.994889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.996658 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.996670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.998461 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.998467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.998670 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.998704 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.998774 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.998909 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":26}
I20260925 14:59:03.999670 12558 tablet_bootstrap.cc:492] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: Bootstrap starting.
I20260925 14:59:04.000241 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.973528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.4.243.193:39633)
0925 14:59:03.992935 (+ 19407us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.992941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.992960 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:59:03.993024 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.993029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.993106 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.993207 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.993726 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.994731 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.997091 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.998294 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.999647 (+ 1353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.999804 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.999854 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.999987 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19282,"thread_start_us":19178,"threads_started":1}
I20260925 14:59:04.000898 12106 tablet_service.cc:1467] Processing CreateTablet for tablet 1547819728da405fab49733d060531dd (DEFAULT_TABLE table=test-table [id=512c10dd926c4f3baa88636097cbcf40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:04.001295 12106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1547819728da405fab49733d060531dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:04.001338 12558 tablet_bootstrap.cc:654] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:04.003397 12558 tablet_bootstrap.cc:492] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: No bootstrap required, opened a new log
I20260925 14:59:04.003531 12558 ts_tablet_manager.cc:1397] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:59:04.004256 12558 raft_consensus.cc:348] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.004487 12558 raft_consensus.cc:374] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:04.004580 12558 raft_consensus.cc:729] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0f5351dab84cf89d80a37e0b7be0ed, State: Initialized, Role: FOLLOWER
I20260925 14:59:04.004858 12558 consensus_queue.cc:260] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.005864 12558 ts_tablet_manager.cc:1428] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:04.009073 12465 heartbeater.cc:503] Master 127.4.243.253:37649 was elected leader, sending a full tablet report...
I20260925 14:59:04.014844 12563 tablet_bootstrap.cc:492] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: Bootstrap starting.
I20260925 14:59:04.015779 12560 tablet_bootstrap.cc:492] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: Bootstrap starting.
I20260925 14:59:04.016767 12563 tablet_bootstrap.cc:654] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:04.017976 12560 tablet_bootstrap.cc:654] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:04.018885 12563 tablet_bootstrap.cc:492] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: No bootstrap required, opened a new log
I20260925 14:59:04.019034 12563 ts_tablet_manager.cc:1397] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:59:04.019661 12563 raft_consensus.cc:348] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.019847 12563 raft_consensus.cc:374] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:04.019943 12563 raft_consensus.cc:729] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08a3dae84034ac2aba9be8b32c40894, State: Initialized, Role: FOLLOWER
I20260925 14:59:04.020174 12563 consensus_queue.cc:260] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.020952 12563 ts_tablet_manager.cc:1428] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:59:04.023731 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.006030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.195:42553)
0925 14:59:04.018177 (+ 12147us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.023447 (+ 5270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11993,"thread_start_us":11863,"threads_started":1}
I20260925 14:59:04.024847 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.024338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.195:46989)
0925 14:59:04.024548 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.024659 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 14:59:04.024888 12173 heartbeater.cc:503] Master 127.4.243.253:37649 was elected leader, sending a full tablet report...
I20260925 14:59:04.025223 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.021167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.193:48819)
0925 14:59:04.024815 (+ 3648us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.024987 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3508,"thread_start_us":3440,"threads_started":1}
I20260925 14:59:04.026276 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.025871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.193:60665)
0925 14:59:04.026076 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.026153 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 14:59:04.026930 12560 tablet_bootstrap.cc:492] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: No bootstrap required, opened a new log
I20260925 14:59:04.027071 12560 ts_tablet_manager.cc:1397] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260925 14:59:04.027652 12560 raft_consensus.cc:348] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.027855 12560 raft_consensus.cc:374] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:04.027971 12560 raft_consensus.cc:729] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e7fdf2d9774f679abcb8c82993ec2b, State: Initialized, Role: FOLLOWER
I20260925 14:59:04.028208 12560 consensus_queue.cc:260] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.028868 12560 ts_tablet_manager.cc:1428] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:04.030136 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.029164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.194:36467)
0925 14:59:04.029727 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.029883 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260925 14:59:04.031001 12323 heartbeater.cc:503] Master 127.4.243.253:37649 was elected leader, sending a full tablet report...
I20260925 14:59:04.031075 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.030711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.194:43595)
0925 14:59:04.030843 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.030925 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:59:04.404945 12568 raft_consensus.cc:482] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:04.405143 12568 raft_consensus.cc:504] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.406630 12568 leader_election.cc:288] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633), db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:59:04.413189 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.407077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:55953 (local address 127.4.243.195:41717)
0925 14:59:04.407266 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:04.407272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:04.407293 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:59:04.407463 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:04.407470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:04.407558 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:04.407621 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:04.408293 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.409002 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.411240 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.412396 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.412509 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.412559 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:04.412927 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 14:59:04.413011 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:59:04.413314 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.406933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:41717 (local address 127.4.243.194:55953)
0925 14:59:04.407096 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.407118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:04.407278 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:04.407825 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:04.407828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:04.407848 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:04.408150 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:04.408158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.409188 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.409193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:04.411096 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:04.411107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.412557 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.412563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:04.412757 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.412792 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:59:04.412900 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:59:04.413037 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:59:04.414021 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1547819728da405fab49733d060531dd" candidate_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" is_pre_election: true
I20260925 14:59:04.414345 12418 raft_consensus.cc:2454] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2e7fdf2d9774f679abcb8c82993ec2b in term 0.
I20260925 14:59:04.414419 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.407651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:43083 (local address 127.4.243.193:39633)
0925 14:59:04.407847 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:04.407855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:04.407877 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:59:04.408084 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:04.408089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:04.408153 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:04.408247 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:04.409067 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.410104 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.412284 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.413423 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.413586 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.413657 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:04.414011 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 14:59:04.414157 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:59:04.414484 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.406933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.4.243.194:43083)
0925 14:59:04.407761 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.407778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:04.407946 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:04.408578 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:04.408581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:04.408599 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:04.408923 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:04.408933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.410274 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.410282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:04.412138 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:04.412152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.413683 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.413687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:04.413816 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.413873 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 14:59:04.413986 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:59:04.414100 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":143,"thread_start_us":334,"threads_started":1}
I20260925 14:59:04.414898 12211 leader_election.cc:302] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0f5351dab84cf89d80a37e0b7be0ed, f2e7fdf2d9774f679abcb8c82993ec2b; no voters:
I20260925 14:59:04.415351 12568 raft_consensus.cc:2781] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:04.415477 12568 raft_consensus.cc:482] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:04.415567 12568 raft_consensus.cc:3037] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:04.415393 12126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1547819728da405fab49733d060531dd" candidate_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08a3dae84034ac2aba9be8b32c40894" is_pre_election: true
I20260925 14:59:04.415778 12126 raft_consensus.cc:2454] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2e7fdf2d9774f679abcb8c82993ec2b in term 0.
I20260925 14:59:04.417869 12568 raft_consensus.cc:504] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.418478 12568 leader_election.cc:288] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [CANDIDATE]: Term 1 election: Requested vote from peers b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633), db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717)
I20260925 14:59:04.418943 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1547819728da405fab49733d060531dd" candidate_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0f5351dab84cf89d80a37e0b7be0ed"
I20260925 14:59:04.418957 12126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1547819728da405fab49733d060531dd" candidate_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08a3dae84034ac2aba9be8b32c40894"
I20260925 14:59:04.419145 12126 raft_consensus.cc:3037] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:04.419137 12418 raft_consensus.cc:3037] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:04.421155 12418 raft_consensus.cc:2454] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2e7fdf2d9774f679abcb8c82993ec2b in term 1.
I20260925 14:59:04.421289 12126 raft_consensus.cc:2454] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2e7fdf2d9774f679abcb8c82993ec2b in term 1.
I20260925 14:59:04.421516 12211 leader_election.cc:302] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0f5351dab84cf89d80a37e0b7be0ed, f2e7fdf2d9774f679abcb8c82993ec2b; no voters:
I20260925 14:59:04.421816 12568 raft_consensus.cc:2781] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:04.422250 12568 raft_consensus.cc:686] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 LEADER]: Becoming Leader. State: Replica: f2e7fdf2d9774f679abcb8c82993ec2b, State: Running, Role: LEADER
I20260925 14:59:04.422690 12568 consensus_queue.cc:237] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } }
I20260925 14:59:04.423280 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.422832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.194:34597)
0925 14:59:04.423022 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.423153 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:59:04.426154 11805 catalog_manager.cc:5495] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b reported cstate change: term changed from 0 to 1, leader changed from <none> to f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194). New cstate: current_term: 1 leader_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2e7fdf2d9774f679abcb8c82993ec2b" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46197 } health_report { overall_health: HEALTHY } } }
I20260925 14:59:04.446143 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.440784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.4.243.194:46197)
0925 14:59:04.441030 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:04.441037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:04.441253 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:59:04.441501 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:04.441506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:04.441611 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:04.441766 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:04.442127 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.442919 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.444142 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.444531 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.444661 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.445066 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:04.445584 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:04.445742 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:04.445797 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:59:04.445868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:59:04.449944 12418 raft_consensus.cc:1261] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Refusing update from remote peer f2e7fdf2d9774f679abcb8c82993ec2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:04.450306 12126 raft_consensus.cc:1261] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 1 FOLLOWER]: Refusing update from remote peer f2e7fdf2d9774f679abcb8c82993ec2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:04.450675 12571 consensus_queue.cc:1035] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0f5351dab84cf89d80a37e0b7be0ed" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:04.451155 12568 consensus_queue.cc:1035] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b08a3dae84034ac2aba9be8b32c40894" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:04.453409 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.452984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.195:42161)
0925 14:59:04.453121 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.453233 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:59:04.454391 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.453712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.193:59647)
0925 14:59:04.454009 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.454144 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260925 14:59:04.459525 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.459123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.193:58471)
0925 14:59:04.459266 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.459361 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:59:04.462297 12575 mvcc.cc:205] Tried to move back new op lower bound from 7333266818860552192 to 7333266818759544832. Current Snapshot: MvccSnapshot[applied={T|T < 7333266818860552192}]
I20260925 14:59:04.463670 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.463166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.195:43293)
0925 14:59:04.463342 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.463455 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:59:04.472885 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.472509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.4.243.194:58893)
0925 14:59:04.472699 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.472764 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 14:59:04.481665 11803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:04.481926 11803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:04.487241 11803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2492809c32e7400588ed011c799620e5: Sending DeleteTablet for 3 replicas of tablet 1547819728da405fab49733d060531dd
I20260925 14:59:04.488284 12106 tablet_service.cc:1514] Processing DeleteTablet for tablet 1547819728da405fab49733d060531dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:04 UTC) from {username='oryx', principal='oryx/127.4.243.253@KRBTEST.COM'} at 127.0.0.1:42578
I20260925 14:59:04.488300 12398 tablet_service.cc:1514] Processing DeleteTablet for tablet 1547819728da405fab49733d060531dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:04 UTC) from {username='oryx', principal='oryx/127.4.243.253@KRBTEST.COM'} at 127.0.0.1:48062
I20260925 14:59:04.488917 12585 tablet_replica.cc:330] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: stopping tablet replica
I20260925 14:59:04.489099 12585 raft_consensus.cc:2229] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:04.489223 12256 tablet_service.cc:1514] Processing DeleteTablet for tablet 1547819728da405fab49733d060531dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:04 UTC) from {username='oryx', principal='oryx/127.4.243.253@KRBTEST.COM'} at 127.0.0.1:43636
I20260925 14:59:04.489378 12585 raft_consensus.cc:2258] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:04.490448 12585 ts_tablet_manager.cc:1905] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:04.496361 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11981
I20260925 14:59:04.497236 12585 ts_tablet_manager.cc:1918] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:04.497365 12585 log.cc:1196] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/1547819728da405fab49733d060531dd
I20260925 14:59:04.497785 12585 ts_tablet_manager.cc:1939] T 1547819728da405fab49733d060531dd P db0f5351dab84cf89d80a37e0b7be0ed: Deleting consensus metadata
I20260925 14:59:04.498894 11791 catalog_manager.cc:4841] TS db0f5351dab84cf89d80a37e0b7be0ed (127.4.243.195:41717): tablet 1547819728da405fab49733d060531dd (table test-table [id=512c10dd926c4f3baa88636097cbcf40]) successfully deleted
I20260925 14:59:04.503182 12587 tablet_replica.cc:330] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: stopping tablet replica
I20260925 14:59:04.503433 12587 raft_consensus.cc:2229] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:04.504086 12587 raft_consensus.cc:2258] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:04.504731 12586 tablet_replica.cc:330] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: stopping tablet replica
I20260925 14:59:04.504930 12586 raft_consensus.cc:2229] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:04.505208 12586 raft_consensus.cc:2258] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:04.505976 12587 ts_tablet_manager.cc:1905] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:04.513048 12587 ts_tablet_manager.cc:1918] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:04.513224 12587 log.cc:1196] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/1547819728da405fab49733d060531dd
I20260925 14:59:04.513723 12587 ts_tablet_manager.cc:1939] T 1547819728da405fab49733d060531dd P f2e7fdf2d9774f679abcb8c82993ec2b: Deleting consensus metadata
I20260925 14:59:04.515029 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.503205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37616)
0925 14:59:04.514756 (+ 11551us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.514865 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11430,"thread_start_us":11334,"threads_started":1}
I20260925 14:59:04.515605 12586 ts_tablet_manager.cc:1905] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:04.519838 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.497805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.4.243.193:39633)
0925 14:59:04.498021 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:04.498030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:04.498779 (+ 749us) server_negotiation.cc:407] Connection header received
0925 14:59:04.498863 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:04.498871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:04.498972 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:04.499066 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:04.503898 (+ 4832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.505116 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.507638 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.508286 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.508390 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.512095 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:04.512111 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:04.512132 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:04.512172 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:04.516307 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.517113 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.517144 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.517148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.517293 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.517868 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.517874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.517877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.518341 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:04.518650 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:04.519258 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 14:59:04.519498 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:59:04.521252 12166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:04.523049 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.522702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37618)
0925 14:59:04.522849 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.522929 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:59:04.525759 11791 catalog_manager.cc:4841] TS f2e7fdf2d9774f679abcb8c82993ec2b (127.4.243.194:46197): tablet 1547819728da405fab49733d060531dd (table test-table [id=512c10dd926c4f3baa88636097cbcf40]) successfully deleted
I20260925 14:59:04.637650 12586 ts_tablet_manager.cc:1918] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:04.637889 12586 log.cc:1196] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/1547819728da405fab49733d060531dd
I20260925 14:59:04.638293 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11981
I20260925 14:59:04.638422 12586 ts_tablet_manager.cc:1939] T 1547819728da405fab49733d060531dd P b08a3dae84034ac2aba9be8b32c40894: Deleting consensus metadata
I20260925 14:59:04.666113 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 12186
I20260925 14:59:04.666200 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.665729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42594)
0925 14:59:04.665932 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.666022 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 14:59:04.666821 11792 catalog_manager.cc:4584] TS b08a3dae84034ac2aba9be8b32c40894 (127.4.243.193:39633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1547819728da405fab49733d060531dd: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
I20260925 14:59:04.682883 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.667185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.4.243.194:46197)
0925 14:59:04.667434 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:04.667443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:04.667464 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:59:04.667521 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:04.667526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:04.667626 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:04.667723 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:04.669532 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.670710 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.672523 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.673141 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.673273 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.676403 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:04.676425 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:04.676445 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:04.676484 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:04.680277 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.680989 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.680995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.680997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.681121 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.681464 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.681467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.681469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.681779 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:04.682071 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:04.682305 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 14:59:04.682460 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 14:59:04.683575 12316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:04.727272 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.726753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42610)
0925 14:59:04.726979 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.727098 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 14:59:04.728291 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.727907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42614)
0925 14:59:04.728062 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.728143 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 14:59:04.797050 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.796506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42616)
0925 14:59:04.796751 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.796859 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 14:59:04.797704 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 12186
I20260925 14:59:04.798341 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.797814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42622)
0925 14:59:04.798073 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.798170 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260925 14:59:04.817667 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 12334
I20260925 14:59:04.830655 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.818330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.4.243.195:41717)
0925 14:59:04.818508 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:04.818515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:04.818565 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:59:04.818810 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:04.818814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:04.818873 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:04.818978 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:04.820330 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.821078 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.822260 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.822785 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.822916 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.825226 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:04.825249 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:04.825267 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:04.825308 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:04.828010 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.828642 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.828657 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.828659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.828775 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.829153 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.829158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.829162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.829502 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:04.829796 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:04.830105 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 14:59:04.830303 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:59:04.831271 12458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:04.895228 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.894787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37622)
0925 14:59:04.894986 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.895089 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 14:59:04.896317 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.895934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38093 (local address 127.0.0.1:37630)
0925 14:59:04.896126 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.896193 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 14:59:04.900391 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.900067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42626)
0925 14:59:04.900231 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.900294 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 14:59:04.901198 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.900888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39633 (local address 127.0.0.1:42632)
0925 14:59:04.901010 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:04.901072 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:59:04.922693 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 12334
I20260925 14:59:04.940348 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11668
I20260925 14:59:04.941371 11835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:05.059681 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11668
I20260925 14:59:05.080371 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 11769
I20260925 14:59:05.081442 12024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:05.192674 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 11769
2026-09-25T14:59:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9663 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:05 dist-test-slave-qxh0 krb5kdc[12598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:05 dist-test-slave-qxh0 krb5kdc[12598](info): set up 2 sockets
Sep 25 14:59:05 dist-test-slave-qxh0 krb5kdc[12598](info): commencing operation
krb5kdc: starting...
W20260925 14:59:06.275235 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:59:06 dist-test-slave-qxh0 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab.
Entry for principal oryx/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab.
I20260925 14:59:06.470873 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37443
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab
--principal=oryx/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:06.776618 12611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:06.776933 12611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:06.777093 12611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:06.783972 12611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:06.784087 12611 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:59:06.784168 12611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:06.784272 12611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:06.784363 12611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:06.784463 12611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:06.793610 12611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:37443
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.254.keytab
--principal=oryx/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:06.794761 12611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:06.796016 12611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:06.807226 12617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:06.807181 12619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:06.807492 12616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:06.808229 12611 server_base.cc:1034] running on GCE node
I20260925 14:59:06.808949 12611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:06.810242 12611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:06.811494 12611 hybrid_clock.cc:648] HybridClock initialized: now 1790348346811428 us; error 40 us; skew 500 ppm
I20260925 14:59:06.811810 12611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:06 dist-test-slave-qxh0 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348346, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:06.814857 12611 init.cc:377] Logged in from keytab as oryx/127.4.243.254@KRBTEST.COM (short username oryx)
I20260925 14:59:06.816203 12611 webserver.cc:460] Webserver started at http://127.4.243.254:45237/ using document root <none> and password file <none>
I20260925 14:59:06.816821 12611 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:06.816903 12611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:06.817169 12611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:06.820631 12611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "4e863cf769e64c4fad86a1907d5b63c6"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-qxh0"
server_key: "1ac9cbcca4de0a953a7fa79f7853cfe4"
server_key_iv: "c43ef78599d0b02f1e9a9c4e6eac5bab"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.821637 12611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "4e863cf769e64c4fad86a1907d5b63c6"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-qxh0"
server_key: "1ac9cbcca4de0a953a7fa79f7853cfe4"
server_key_iv: "c43ef78599d0b02f1e9a9c4e6eac5bab"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.826402 12611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 14:59:06.829630 12626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.830941 12611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:06.831105 12611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "4e863cf769e64c4fad86a1907d5b63c6"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-qxh0"
server_key: "1ac9cbcca4de0a953a7fa79f7853cfe4"
server_key_iv: "c43ef78599d0b02f1e9a9c4e6eac5bab"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.831292 12611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:06.846107 12611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:06.850067 12611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:06.850325 12611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:06.871037 12611 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:37443
I20260925 14:59:06.871061 12687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:37443 every 8 connection(s)
I20260925 14:59:06.872314 12611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:06.875764 12688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:06.878731 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 12611
I20260925 14:59:06.878978 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:59:06.879751 5071 external_mini_cluster.cc:1428] Setting key 30e3e1e68ef420bf10558db55279e5ce
I20260925 14:59:06.888001 12688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Bootstrap starting.
I20260925 14:59:06.891615 12688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:06.892877 12688 log.cc:824] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Log is configured to *not* fsync() on all Append() calls
Sep 25 14:59:06 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:06.895884 12688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: No bootstrap required, opened a new log
I20260925 14:59:06.902956 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.881658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.4.243.254:37443)
0925 14:59:06.882357 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.882368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.882406 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:59:06.884107 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.884127 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.884492 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.884816 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:06.886521 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.887743 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.889483 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.890109 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.890226 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.894917 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:06.894940 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:06.894957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:06.895015 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:06.898798 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:06.899385 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:06.899392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:06.899400 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:06.899603 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:06.899950 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:06.899955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:06.899959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:06.900627 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:06.900866 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.901245 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.901672 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":335,"threads_started":1}
I20260925 14:59:06.905799 12688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e863cf769e64c4fad86a1907d5b63c6" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37443 } }
I20260925 14:59:06.906080 12688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:06.906164 12688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e863cf769e64c4fad86a1907d5b63c6, State: Initialized, Role: FOLLOWER
I20260925 14:59:06.906670 12688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e863cf769e64c4fad86a1907d5b63c6" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37443 } }
I20260925 14:59:06.906864 12688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:06.906955 12688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:06.907061 12688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:06.908948 12688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e863cf769e64c4fad86a1907d5b63c6" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37443 } }
I20260925 14:59:06.909355 12688 leader_election.cc:302] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e863cf769e64c4fad86a1907d5b63c6; no voters:
I20260925 14:59:06.909929 12688 leader_election.cc:288] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:06.910055 12693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:06.910761 12693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [term 1 LEADER]: Becoming Leader. State: Replica: 4e863cf769e64c4fad86a1907d5b63c6, State: Running, Role: LEADER
I20260925 14:59:06.911276 12693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e863cf769e64c4fad86a1907d5b63c6" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37443 } }
I20260925 14:59:06.912127 12688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:06.914161 12694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e863cf769e64c4fad86a1907d5b63c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e863cf769e64c4fad86a1907d5b63c6" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37443 } } }
I20260925 14:59:06.914244 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e863cf769e64c4fad86a1907d5b63c6. Latest consensus state: current_term: 1 leader_uuid: "4e863cf769e64c4fad86a1907d5b63c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e863cf769e64c4fad86a1907d5b63c6" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37443 } } }
I20260925 14:59:06.914651 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:06.914716 12694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:06.916186 12698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:06.921932 12698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:06.930332 12698 catalog_manager.cc:1347] Generated new cluster ID: 32bac6cfe8fa406cad35a455b834dad2
I20260925 14:59:06.930447 12698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:06.948252 12698 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:06.949483 12698 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:06.964388 12698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Generated new TSK 0
I20260925 14:59:06.965224 12698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab.
Entry for principal oryx/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab.
I20260925 14:59:07.012099 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37443
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab
--principal=oryx/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:07.327122 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:07.327381 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:07.327476 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:07.334623 12714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:07.334738 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:07.334817 12714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:59:07.334895 12714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:59:07.344298 12714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab
--principal=oryx/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:07.345485 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:07.346789 12714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:07.359436 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:07.359467 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:07.359493 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:07.360421 12714 server_base.cc:1034] running on GCE node
I20260925 14:59:07.360915 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:07.361712 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:07.362938 12714 hybrid_clock.cc:648] HybridClock initialized: now 1790348347362882 us; error 37 us; skew 500 ppm
I20260925 14:59:07.363224 12714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:07 dist-test-slave-qxh0 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348347, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:07.366173 12714 init.cc:377] Logged in from keytab as oryx/127.4.243.193@KRBTEST.COM (short username oryx)
I20260925 14:59:07.367558 12714 webserver.cc:460] Webserver started at http://127.4.243.193:37327/ using document root <none> and password file <none>
I20260925 14:59:07.368160 12714 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:07.368242 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:07.368485 12714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:07.371268 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "699b62e4b4c24853bddfa2d1126bfd21"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-qxh0"
server_key: "07b0ad7d6b6048038399e1c150db0900"
server_key_iv: "8f30b4171816d51ef9acf97702e10adf"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.371963 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "699b62e4b4c24853bddfa2d1126bfd21"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-qxh0"
server_key: "07b0ad7d6b6048038399e1c150db0900"
server_key_iv: "8f30b4171816d51ef9acf97702e10adf"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.376691 12714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:59:07.379853 12729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:07.381067 12714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:07.381217 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "699b62e4b4c24853bddfa2d1126bfd21"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-qxh0"
server_key: "07b0ad7d6b6048038399e1c150db0900"
server_key_iv: "8f30b4171816d51ef9acf97702e10adf"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.381418 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:07.408514 12714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:07.412273 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:07.412566 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:07.415259 12714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:07.415364 12714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:07.415509 12714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:07.415571 12714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:59:07 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348347, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:07.439226 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.415548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:38243 (local address 127.4.243.254:37443)
0925 14:59:07.416270 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:07.416277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:07.417195 (+ 918us) server_negotiation.cc:407] Connection header received
0925 14:59:07.419384 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:07.419390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:07.419470 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:07.419599 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:07.421451 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.422514 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.425583 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.425972 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.426098 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.430832 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:07.430840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:07.430848 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:07.430901 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:07.435140 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.436393 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.436400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.436406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.436559 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.437079 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.437087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.437093 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.437555 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:07.437742 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:07.438391 (+ 649us) server_negotiation.cc:299] Negotiation successful
0925 14:59:07.438706 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":521,"threads_started":1}
I20260925 14:59:07.439808 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.416130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37443 (local address 127.4.243.193:38243)
0925 14:59:07.417037 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:07.417085 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:07.419081 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:07.419815 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:07.419834 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:07.420451 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:07.421255 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.421283 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.422703 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.422709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.425422 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.425433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.426235 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.426244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.426459 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.427189 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:07.427226 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:07.430488 (+ 3262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:07.435374 (+ 4886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.435386 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.435413 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.436180 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.436749 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.436753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.436757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.436952 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.437698 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:07.437703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:07.438018 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 14:59:07.438229 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 14:59:07.438641 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":413,"threads_started":1}
I20260925 14:59:07.459221 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.445170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:50649 (local address 127.4.243.254:37443)
0925 14:59:07.445386 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:07.445394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:07.445908 (+ 514us) server_negotiation.cc:407] Connection header received
0925 14:59:07.446091 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:07.446094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:07.446146 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:07.446217 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:07.447755 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.448648 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.450194 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.450710 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.450855 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.453066 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:07.453074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:07.453082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:07.453160 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:07.456427 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.457346 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.457352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.457356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.457479 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.457831 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.457837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.457843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.458103 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:07.458250 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:07.458657 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 14:59:07.458876 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:59:07.459376 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.445094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37443 (local address 127.4.243.193:50649)
0925 14:59:07.445793 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:07.445816 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:07.445971 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:07.446277 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:07.446284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:07.447002 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:07.447577 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.447595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.448839 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.448846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.450051 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.450063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.450916 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.450922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.451081 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.451716 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:07.451743 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:07.452779 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:07.456549 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.456558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.456562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.457205 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.457592 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.457596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.457600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.457730 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.458216 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:07.458221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:07.458392 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 14:59:07.458531 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 14:59:07.458906 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":348,"threads_started":1}
I20260925 14:59:07.473052 12714 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:43061
I20260925 14:59:07.473090 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:43061 every 8 connection(s)
I20260925 14:59:07.474210 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:07.476162 12850 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37443
I20260925 14:59:07.476377 12850 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:07.476981 12850 heartbeater.cc:511] Master 127.4.243.254:37443 requested a full tablet report, sending...
I20260925 14:59:07.478922 12643 ts_manager.cc:194] Registered new tserver with Master: 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061)
I20260925 14:59:07.480257 12643 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:38243
I20260925 14:59:07.482529 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 12714
I20260925 14:59:07.482758 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:07.483337 5071 external_mini_cluster.cc:1428] Setting key 2d9a8757414a6229a9b3cbeb7af1232a
WARNING: no policy specified for oryx/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab.
Entry for principal oryx/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab.
I20260925 14:59:07.526127 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37443
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab
--principal=oryx/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:07.838716 12854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:07.838975 12854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:07.839066 12854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:07.846093 12854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:07.846191 12854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:07.846271 12854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:59:07.846351 12854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:59:07.855594 12854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab
--principal=oryx/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:07.856714 12854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:07.857961 12854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:07.870301 12859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:07.871045 12854 server_base.cc:1034] running on GCE node
W20260925 14:59:07.873445 12862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:07.873418 12860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:07.874156 12854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:07.875144 12854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:07.876376 12854 hybrid_clock.cc:648] HybridClock initialized: now 1790348347876309 us; error 43 us; skew 500 ppm
I20260925 14:59:07.876648 12854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:07 dist-test-slave-qxh0 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348347, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:07.879711 12854 init.cc:377] Logged in from keytab as oryx/127.4.243.194@KRBTEST.COM (short username oryx)
I20260925 14:59:07.881027 12854 webserver.cc:460] Webserver started at http://127.4.243.194:35791/ using document root <none> and password file <none>
I20260925 14:59:07.881651 12854 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:07.881729 12854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:07.882081 12854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:07.884917 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "a55b8b422e014257be586b950ebdc363"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-qxh0"
server_key: "a9ba26c3a80470ab15bd5cbaf6790e8b"
server_key_iv: "80c64b601deeacebad97396d96f8b12f"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.885805 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "a55b8b422e014257be586b950ebdc363"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-qxh0"
server_key: "a9ba26c3a80470ab15bd5cbaf6790e8b"
server_key_iv: "80c64b601deeacebad97396d96f8b12f"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.890810 12854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:59:07.894153 12869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:07.895427 12854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:07.895584 12854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "a55b8b422e014257be586b950ebdc363"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-qxh0"
server_key: "a9ba26c3a80470ab15bd5cbaf6790e8b"
server_key_iv: "80c64b601deeacebad97396d96f8b12f"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.895789 12854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:07.906244 12854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:07.910305 12854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:07.910557 12854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:07.913007 12854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:07.913115 12854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:07.913257 12854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:07.913354 12854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:59:07 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348347, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:07.940701 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.913718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:52899 (local address 127.4.243.254:37443)
0925 14:59:07.913918 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:07.913925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:07.915610 (+ 1685us) server_negotiation.cc:407] Connection header received
0925 14:59:07.918864 (+ 3254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:07.918874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:07.919042 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:07.919180 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:07.922434 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.923911 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.928060 (+ 4149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.928806 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.928941 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.933689 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:07.933695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:07.933698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:07.933729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:07.937405 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.938304 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.938310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.938315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.938476 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.939050 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.939055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.939059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.939360 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:07.939502 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:07.940169 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 14:59:07.940371 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 14:59:07.941927 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.914420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37443 (local address 127.4.243.194:52899)
0925 14:59:07.915449 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:07.915495 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:07.917891 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:07.919576 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:07.919596 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:07.920910 (+ 1314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:07.922164 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.922196 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.924293 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.924299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.927850 (+ 3551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.927866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.929039 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.929046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.929307 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.930132 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:07.930155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:07.933381 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:07.937546 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.937558 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.937579 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.938165 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.938670 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.938675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.938679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.938918 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.939495 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:07.939503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:07.939833 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 14:59:07.940085 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 14:59:07.940601 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":470,"threads_started":1}
I20260925 14:59:07.965162 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.948259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:52149 (local address 127.4.243.254:37443)
0925 14:59:07.948473 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:07.948481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:07.951562 (+ 3081us) server_negotiation.cc:407] Connection header received
0925 14:59:07.951745 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:07.951767 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:07.951835 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:07.951913 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:07.953304 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.953971 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.956450 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.956843 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.956936 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.959440 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:07.959448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:07.959455 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:07.959497 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:07.962146 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.962856 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.962861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.962865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.963041 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.963544 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.963552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.963558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.964002 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:07.964151 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:07.964575 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 14:59:07.964842 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:59:07.965253 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.947856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37443 (local address 127.4.243.194:52149)
0925 14:59:07.951447 (+ 3591us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:07.951471 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:07.951627 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:07.951979 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:07.951986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:07.952669 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:07.953135 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.953151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.954123 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.954131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.956289 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:07.956306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.957042 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.957054 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:07.957279 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.958027 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:07.958043 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:07.959253 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:07.962279 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.962285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.962289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.962730 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.963170 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:07.963179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:07.963185 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:07.963381 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:07.964122 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:07.964127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:07.964315 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 14:59:07.964454 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 14:59:07.964768 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3396,"thread_start_us":3245,"threads_started":1}
I20260925 14:59:07.973867 12854 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:42121
I20260925 14:59:07.973958 12988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:42121 every 8 connection(s)
I20260925 14:59:07.975148 12854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:07.976528 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 12854
I20260925 14:59:07.976742 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:07.977182 12989 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37443
I20260925 14:59:07.977276 5071 external_mini_cluster.cc:1428] Setting key 83900ce9822e5a813f977690dc5324a1
I20260925 14:59:07.977399 12989 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:07.978133 12989 heartbeater.cc:511] Master 127.4.243.254:37443 requested a full tablet report, sending...
I20260925 14:59:07.979509 12643 ts_manager.cc:194] Registered new tserver with Master: a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121)
I20260925 14:59:07.980685 12643 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:52899
WARNING: no policy specified for oryx/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab.
Entry for principal oryx/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab.
I20260925 14:59:08.022001 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37443
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab
--principal=oryx/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:08.340054 12993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:08.340341 12993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:08.340440 12993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:08.347749 12993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:08.347862 12993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:08.347945 12993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:59:08.348021 12993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:59:08.357515 12993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab
--principal=oryx/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.243.254:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:08.358647 12993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:08.359920 12993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:08.372215 12998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:08.372200 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:08.372246 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:08.372977 12993 server_base.cc:1034] running on GCE node
I20260925 14:59:08.373621 12993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:08.374527 12993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:08.375831 12993 hybrid_clock.cc:648] HybridClock initialized: now 1790348348375762 us; error 33 us; skew 500 ppm
I20260925 14:59:08.376132 12993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:08 dist-test-slave-qxh0 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:08.379380 12993 init.cc:377] Logged in from keytab as oryx/127.4.243.195@KRBTEST.COM (short username oryx)
I20260925 14:59:08.380586 12993 webserver.cc:460] Webserver started at http://127.4.243.195:37623/ using document root <none> and password file <none>
I20260925 14:59:08.381211 12993 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:08.381289 12993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:08.381527 12993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:08.384445 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "1c569ea0a87b4bf0971de7f6aa60479a"
format_stamp: "Formatted at 2026-09-25 14:59:08 on dist-test-slave-qxh0"
server_key: "56605941c0a70d2cc0780e56af80299c"
server_key_iv: "944606bd1730fe40f9d337efd06a74b5"
server_key_version: "encryptionkey@0"
I20260925 14:59:08.385210 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "1c569ea0a87b4bf0971de7f6aa60479a"
format_stamp: "Formatted at 2026-09-25 14:59:08 on dist-test-slave-qxh0"
server_key: "56605941c0a70d2cc0780e56af80299c"
server_key_iv: "944606bd1730fe40f9d337efd06a74b5"
server_key_version: "encryptionkey@0"
I20260925 14:59:08.390305 12993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 14:59:08.393500 13008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:08.394820 12993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 14:59:08.394938 12993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "1c569ea0a87b4bf0971de7f6aa60479a"
format_stamp: "Formatted at 2026-09-25 14:59:08 on dist-test-slave-qxh0"
server_key: "56605941c0a70d2cc0780e56af80299c"
server_key_iv: "944606bd1730fe40f9d337efd06a74b5"
server_key_version: "encryptionkey@0"
I20260925 14:59:08.395097 12993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:08.435395 12993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:08.439707 12993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:08.439931 12993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:08.442794 12993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:08.442869 12993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:08.443027 12993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:08.443105 12993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:59:08 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:08.468130 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:08.443240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:38151 (local address 127.4.243.254:37443)
0925 14:59:08.443438 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:08.443444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:08.445071 (+ 1627us) server_negotiation.cc:407] Connection header received
0925 14:59:08.448128 (+ 3057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:08.448135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:08.448231 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:08.448356 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:08.450957 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.452011 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.453634 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.454250 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.454401 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:08.459828 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:08.459836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:08.459843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:08.459887 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:08.463762 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:08.464753 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:08.464763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:08.464767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:08.464905 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:08.465528 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:08.465533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:08.465535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:08.465860 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:08.465987 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:08.467413 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0925 14:59:08.467681 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:59:08.470069 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:08.443857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37443 (local address 127.4.243.195:38151)
0925 14:59:08.444874 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:08.444936 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:08.447732 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:08.448826 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:08.448842 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:08.449728 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:08.450741 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:08.450775 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.452156 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.452162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:08.453473 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:08.453485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.454541 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.454557 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:08.454991 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:08.455805 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:08.455854 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:08.459495 (+ 3641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:08.463947 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:08.463960 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:08.463983 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:08.464619 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:08.465036 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:08.465047 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:08.465054 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:08.465382 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:08.466010 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:08.466023 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:08.466630 (+ 607us) client_negotiation.cc:769] Sending connection context
0925 14:59:08.467026 (+ 396us) client_negotiation.cc:240] Negotiation successful
0925 14:59:08.467736 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":424,"threads_started":1}
I20260925 14:59:08.482710 12850 heartbeater.cc:503] Master 127.4.243.254:37443 was elected leader, sending a full tablet report...
I20260925 14:59:08.491585 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:08.476694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:44595 (local address 127.4.243.254:37443)
0925 14:59:08.476987 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:08.476996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:08.477587 (+ 591us) server_negotiation.cc:407] Connection header received
0925 14:59:08.477779 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:08.477784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:08.477852 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:08.477955 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:08.479021 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.480251 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.481719 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.482130 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.482269 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:08.484951 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:08.484956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:08.484959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:08.484987 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:08.488566 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:08.489456 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:08.489463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:08.489468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:08.489618 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:08.490055 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:08.490063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:08.490066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:08.490371 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:08.490461 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:08.491020 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 14:59:08.491226 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 14:59:08.491964 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:08.476709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37443 (local address 127.4.243.195:44595)
0925 14:59:08.477489 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:08.477506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:08.477650 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:08.477976 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:08.477981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:08.478427 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:08.478820 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:08.478838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.480438 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.480446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:08.481577 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:08.481588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.482312 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.482318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:08.482495 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:08.483496 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:08.483516 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:08.484668 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:08.488693 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:08.488699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:08.488703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:08.489260 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:08.489751 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:08.489755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:08.489757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:08.489905 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:08.490533 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:08.490536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:08.490746 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 14:59:08.490923 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 14:59:08.491268 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":476,"threads_started":1}
I20260925 14:59:08.495680 12993 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:43923
I20260925 14:59:08.495719 13127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:43923 every 8 connection(s)
I20260925 14:59:08.496829 12993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:08.498739 13128 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37443
I20260925 14:59:08.499006 13128 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:08.499689 13128 heartbeater.cc:511] Master 127.4.243.254:37443 requested a full tablet report, sending...
I20260925 14:59:08.500919 12643 ts_manager.cc:194] Registered new tserver with Master: 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923)
I20260925 14:59:08.501324 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 12993
I20260925 14:59:08.501525 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:08.502027 5071 external_mini_cluster.cc:1428] Setting key 7c4a736bea8d2706ea52247c85aa03b6
I20260925 14:59:08.502063 12643 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:38151
I20260925 14:59:08.504899 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:59:08 dist-test-slave-qxh0 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348348, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:59:08 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:08 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:08.983618 12989 heartbeater.cc:503] Master 127.4.243.254:37443 was elected leader, sending a full tablet report...
I20260925 14:59:09.504583 13128 heartbeater.cc:503] Master 127.4.243.254:37443 was elected leader, sending a full tablet report...
W20260925 14:59:09.594632 13136 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.048s user 0.006s sys 0.007s
W20260925 14:59:09.595850 12737 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:08.540729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.4.243.254:37443)
0925 14:59:08.540958 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:08.540965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:08.541271 (+ 306us) server_negotiation.cc:407] Connection header received
0925 14:59:08.541477 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:08.541483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:08.541562 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:08.541665 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:08.543077 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.543821 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.545129 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:08.545541 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:08.545669 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.595532 (+1049863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43124: BlockingRecv error: recv got EOF from 127.0.0.1:43124 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 14:59:09 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348348, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:09.616846 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.600258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.4.243.254:37443)
0925 14:59:09.600426 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.600431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.600750 (+ 319us) server_negotiation.cc:407] Connection header received
0925 14:59:09.600892 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.600896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.600979 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.601069 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:09.602128 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.602899 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.604196 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.604593 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.604692 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.611850 (+ 7158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:09.611855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:09.611859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:09.611890 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:09.614547 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:09.615320 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:09.615324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:09.615326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:09.615450 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:09.615771 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:09.615774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:09.615776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:09.616053 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:09.616268 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.616463 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.616628 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:59:09.620814 12643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:09.627691 12643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:09.656463 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.638781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.4.243.193:43061)
0925 14:59:09.648792 (+ 10011us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.648798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.648845 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:59:09.648941 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.648949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.649192 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.649790 (+ 598us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:09.650467 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.651968 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.654335 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.655377 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.655535 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.655744 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.655987 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.656152 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9815,"spinlock_wait_cycles":22272,"thread_start_us":9788,"threads_started":1}
I20260925 14:59:09.656538 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.638571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:43061 (local address 127.0.0.1:42910)
0925 14:59:09.639736 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:09.639778 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:09.640020 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:09.649404 (+ 9384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:09.649409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:09.649466 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:09.650262 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.650280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.652096 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.652102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.654154 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.654166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.655563 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.655569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.655736 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.655788 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:59:09.655951 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 14:59:09.656134 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":137,"spinlock_wait_cycles":768,"thread_start_us":1994,"threads_started":1}
I20260925 14:59:09.658344 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.639756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.4.243.194:42121)
0925 14:59:09.640557 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.640566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.648745 (+ 8179us) server_negotiation.cc:407] Connection header received
0925 14:59:09.649016 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.649024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.649196 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.649396 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:09.650412 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.651843 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.654773 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.656211 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.656326 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.656525 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.657907 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.658072 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":645,"threads_started":1}
I20260925 14:59:09.658490 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.638769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:43923 (local address 127.0.0.1:40438)
0925 14:59:09.648544 (+ 9775us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:09.648563 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:09.648760 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:09.649654 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:09.649658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:09.649694 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:09.650292 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.650306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.653129 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.653134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.655375 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.655388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.657480 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.657485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.657654 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.657751 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 14:59:09.657838 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:59:09.658030 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9603,"mutex_wait_us":109,"spinlock_wait_cycles":2304,"thread_start_us":9440,"threads_started":1}
I20260925 14:59:09.658706 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.639458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42121 (local address 127.0.0.1:41364)
0925 14:59:09.648652 (+ 9194us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:09.648667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:09.648844 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:09.649394 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:09.649401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:09.649452 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:09.650095 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.650117 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.651982 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.651989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.654581 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.654620 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.657436 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.657445 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.657650 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.657748 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 14:59:09.657838 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:59:09.658030 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9087,"mutex_wait_us":247,"spinlock_wait_cycles":1920,"thread_start_us":8892,"threads_started":1}
I20260925 14:59:09.659202 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.639188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.4.243.195:43923)
0925 14:59:09.648134 (+ 8946us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.648146 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.648651 (+ 505us) server_negotiation.cc:407] Connection header received
0925 14:59:09.648942 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.648949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.649193 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.649384 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:09.650494 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.651983 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.655550 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.656991 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.657102 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.657313 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.658754 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.658930 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8745,"thread_start_us":8572,"threads_started":1}
I20260925 14:59:09.660498 12780 tablet_service.cc:1467] Processing CreateTablet for tablet e3b1e2f26adc4c7a97454c39c9657e77 (DEFAULT_TABLE table=test-table [id=ce67244c78714136832f9e5715eebe76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:09.661690 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b1e2f26adc4c7a97454c39c9657e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:09.661537 12919 tablet_service.cc:1467] Processing CreateTablet for tablet e3b1e2f26adc4c7a97454c39c9657e77 (DEFAULT_TABLE table=test-table [id=ce67244c78714136832f9e5715eebe76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:09.662786 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b1e2f26adc4c7a97454c39c9657e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:09.665177 13058 tablet_service.cc:1467] Processing CreateTablet for tablet e3b1e2f26adc4c7a97454c39c9657e77 (DEFAULT_TABLE table=test-table [id=ce67244c78714136832f9e5715eebe76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:09.666451 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3b1e2f26adc4c7a97454c39c9657e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:09.674367 13153 tablet_bootstrap.cc:492] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Bootstrap starting.
I20260925 14:59:09.677151 13153 tablet_bootstrap.cc:654] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:09.678084 13153 log.cc:824] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:09.680841 13156 tablet_bootstrap.cc:492] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Bootstrap starting.
I20260925 14:59:09.680894 13153 tablet_bootstrap.cc:492] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: No bootstrap required, opened a new log
I20260925 14:59:09.680995 13154 tablet_bootstrap.cc:492] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Bootstrap starting.
I20260925 14:59:09.681143 13153 ts_tablet_manager.cc:1397] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 14:59:09.683874 13154 tablet_bootstrap.cc:654] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:09.684118 13156 tablet_bootstrap.cc:654] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:09.684911 13154 log.cc:824] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:09.685250 13156 log.cc:824] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:09.687700 13156 tablet_bootstrap.cc:492] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: No bootstrap required, opened a new log
I20260925 14:59:09.687863 13154 tablet_bootstrap.cc:492] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: No bootstrap required, opened a new log
I20260925 14:59:09.687978 13156 ts_tablet_manager.cc:1397] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 14:59:09.688103 13154 ts_tablet_manager.cc:1397] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260925 14:59:09.692062 13153 raft_consensus.cc:348] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.692552 13153 raft_consensus.cc:374] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:09.692682 13153 raft_consensus.cc:729] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55b8b422e014257be586b950ebdc363, State: Initialized, Role: FOLLOWER
I20260925 14:59:09.693305 13153 consensus_queue.cc:260] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.694989 13153 ts_tablet_manager.cc:1428] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 14:59:09.695868 13156 raft_consensus.cc:348] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.696254 13156 raft_consensus.cc:374] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:09.696370 13156 raft_consensus.cc:729] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c569ea0a87b4bf0971de7f6aa60479a, State: Initialized, Role: FOLLOWER
I20260925 14:59:09.696954 13156 consensus_queue.cc:260] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.698010 13154 raft_consensus.cc:348] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.698468 13154 raft_consensus.cc:374] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:09.698635 13154 raft_consensus.cc:729] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Initialized, Role: FOLLOWER
I20260925 14:59:09.698712 13156 ts_tablet_manager.cc:1428] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 14:59:09.699409 13154 consensus_queue.cc:260] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.701077 13154 ts_tablet_manager.cc:1428] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 14:59:09.727241 12851 tablet.cc:2369] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:09.727782 12990 tablet.cc:2369] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:09.749105 13129 tablet.cc:2369] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:59:09.750526 13161 raft_consensus.cc:482] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:09.750837 13161 raft_consensus.cc:504] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.752395 13161 leader_election.cc:288] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923), a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121)
I20260925 14:59:09.759083 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.752756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:47911 (local address 127.4.243.195:43923)
0925 14:59:09.752977 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.752984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.753940 (+ 956us) server_negotiation.cc:407] Connection header received
0925 14:59:09.754150 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.754154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.754220 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.754308 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:09.755017 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.755667 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.757431 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.758132 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.758329 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.758413 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.758717 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.758838 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:59:09.759227 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.753007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:38245 (local address 127.4.243.194:42121)
0925 14:59:09.753168 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.753172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.753936 (+ 764us) server_negotiation.cc:407] Connection header received
0925 14:59:09.754150 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.754153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.754220 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.754313 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:09.754975 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.755639 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.757428 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.758298 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.758514 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.758584 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.758870 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.758991 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:59:09.759343 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.752772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:43923 (local address 127.4.243.193:47911)
0925 14:59:09.753826 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:09.753845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:09.754018 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:09.754331 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:09.754335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:09.754391 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:09.754884 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.754893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.755792 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.755800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.757256 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.757270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.758354 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.758361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.758524 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.758546 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:59:09.758694 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 14:59:09.758814 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":43,"spinlock_wait_cycles":34304,"thread_start_us":661,"threads_started":1}
I20260925 14:59:09.759392 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.752860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42121 (local address 127.4.243.193:38245)
0925 14:59:09.753829 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:09.753843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:09.754008 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:09.754329 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:09.754333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:09.754387 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:09.754801 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.754810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.755759 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.755767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.757277 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:09.757288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.758537 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.758543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:09.758716 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.758738 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:59:09.758845 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:59:09.758963 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":56,"spinlock_wait_cycles":62976,"thread_start_us":683,"threads_started":1}
I20260925 14:59:09.760406 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b1e2f26adc4c7a97454c39c9657e77" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" is_pre_election: true
I20260925 14:59:09.760447 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b1e2f26adc4c7a97454c39c9657e77" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363" is_pre_election: true
I20260925 14:59:09.760851 13078 raft_consensus.cc:2454] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 0.
I20260925 14:59:09.760874 12944 raft_consensus.cc:2454] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 0.
I20260925 14:59:09.761474 12732 leader_election.cc:302] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c569ea0a87b4bf0971de7f6aa60479a, 699b62e4b4c24853bddfa2d1126bfd21, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:09.761796 13161 raft_consensus.cc:2781] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:09.761926 13161 raft_consensus.cc:482] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:09.762040 13161 raft_consensus.cc:3037] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:09.764441 13161 raft_consensus.cc:504] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.765079 13161 leader_election.cc:288] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 election: Requested vote from peers 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923), a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121)
I20260925 14:59:09.765378 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b1e2f26adc4c7a97454c39c9657e77" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c569ea0a87b4bf0971de7f6aa60479a"
I20260925 14:59:09.765563 13078 raft_consensus.cc:3037] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:09.765556 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3b1e2f26adc4c7a97454c39c9657e77" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363"
I20260925 14:59:09.765748 12944 raft_consensus.cc:3037] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:09.767728 13078 raft_consensus.cc:2454] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 1.
I20260925 14:59:09.767906 12944 raft_consensus.cc:2454] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 1.
I20260925 14:59:09.768141 12731 leader_election.cc:302] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c569ea0a87b4bf0971de7f6aa60479a, 699b62e4b4c24853bddfa2d1126bfd21; no voters:
I20260925 14:59:09.768453 13161 raft_consensus.cc:2781] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:09.768903 13161 raft_consensus.cc:686] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 LEADER]: Becoming Leader. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Running, Role: LEADER
I20260925 14:59:09.769379 13161 consensus_queue.cc:237] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } }
I20260925 14:59:09.773945 12642 catalog_manager.cc:5495] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "699b62e4b4c24853bddfa2d1126bfd21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:09.805562 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.800331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.4.243.193:43061)
0925 14:59:09.800564 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.800570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.800595 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:59:09.800771 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.800775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.800871 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.800984 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:09.801253 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.801910 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.803262 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.803683 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.803808 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.804126 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:09.804629 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:09.804733 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.805304 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.805368 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 14:59:09.813687 12944 raft_consensus.cc:1261] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Refusing update from remote peer 699b62e4b4c24853bddfa2d1126bfd21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:09.813687 13078 raft_consensus.cc:1261] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Refusing update from remote peer 699b62e4b4c24853bddfa2d1126bfd21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:09.814741 13161 consensus_queue.cc:1035] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:09.815059 13164 consensus_queue.cc:1035] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:09.824575 13167 mvcc.cc:205] Tried to move back new op lower bound from 7333266840826449920 to 7333266840658595840. Current Snapshot: MvccSnapshot[applied={T|T < 7333266840826449920}]
I20260925 14:59:09.826778 13168 mvcc.cc:205] Tried to move back new op lower bound from 7333266840826449920 to 7333266840658595840. Current Snapshot: MvccSnapshot[applied={T|T < 7333266840826449920}]
I20260925 14:59:09.832114 13169 mvcc.cc:205] Tried to move back new op lower bound from 7333266840826449920 to 7333266840658595840. Current Snapshot: MvccSnapshot[applied={T|T < 7333266840826449920}]
I20260925 14:59:09.841382 12642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:09.841624 12642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:09.845237 12642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Sending DeleteTablet for 3 replicas of tablet e3b1e2f26adc4c7a97454c39c9657e77
I20260925 14:59:09.846324 12780 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b1e2f26adc4c7a97454c39c9657e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:09 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:42910
I20260925 14:59:09.846483 13058 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b1e2f26adc4c7a97454c39c9657e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:09 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260925 14:59:09.846696 12919 tablet_service.cc:1514] Processing DeleteTablet for tablet e3b1e2f26adc4c7a97454c39c9657e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:09 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:41364
I20260925 14:59:09.849172 12642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:09.850858 12642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:09.856307 13178 tablet_replica.cc:330] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: stopping tablet replica
I20260925 14:59:09.857337 13178 raft_consensus.cc:2229] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:09.858366 13178 raft_consensus.cc:2258] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:09.860585 13176 tablet_replica.cc:330] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: stopping tablet replica
I20260925 14:59:09.861104 13177 tablet_replica.cc:330] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: stopping tablet replica
I20260925 14:59:09.861863 13176 raft_consensus.cc:2229] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:09.862313 13177 raft_consensus.cc:2229] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:09.862485 12919 tablet_service.cc:1467] Processing CreateTablet for tablet a93f1d556158410da28221ec24c2dcb0 (DEFAULT_TABLE table=test-table [id=78bf4a13e7c64848a0d7e00593bd9db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:09.863044 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93f1d556158410da28221ec24c2dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:09.861987 12780 tablet_service.cc:1467] Processing CreateTablet for tablet a93f1d556158410da28221ec24c2dcb0 (DEFAULT_TABLE table=test-table [id=78bf4a13e7c64848a0d7e00593bd9db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:09.863525 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93f1d556158410da28221ec24c2dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:09.863803 13177 raft_consensus.cc:2258] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:09.864223 13058 tablet_service.cc:1467] Processing CreateTablet for tablet a93f1d556158410da28221ec24c2dcb0 (DEFAULT_TABLE table=test-table [id=78bf4a13e7c64848a0d7e00593bd9db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:09.866933 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93f1d556158410da28221ec24c2dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:09.867326 13178 ts_tablet_manager.cc:1905] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:09.870360 13153 tablet_bootstrap.cc:492] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: Bootstrap starting.
I20260925 14:59:09.872031 13153 tablet_bootstrap.cc:654] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:09.873102 13156 tablet_bootstrap.cc:492] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: Bootstrap starting.
I20260925 14:59:09.874419 13156 tablet_bootstrap.cc:654] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:09.879860 13176 raft_consensus.cc:2258] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:09.881151 13178 ts_tablet_manager.cc:1918] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:09.881428 13178 log.cc:1196] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/e3b1e2f26adc4c7a97454c39c9657e77
I20260925 14:59:09.882102 13177 ts_tablet_manager.cc:1905] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:09.882184 13178 ts_tablet_manager.cc:1939] T e3b1e2f26adc4c7a97454c39c9657e77 P a55b8b422e014257be586b950ebdc363: Deleting consensus metadata
I20260925 14:59:09.885874 12629 catalog_manager.cc:4841] TS a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121): tablet e3b1e2f26adc4c7a97454c39c9657e77 (table test-table [id=ce67244c78714136832f9e5715eebe76]) successfully deleted
I20260925 14:59:09.891806 13177 ts_tablet_manager.cc:1918] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:09.892055 13177 log.cc:1196] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/e3b1e2f26adc4c7a97454c39c9657e77
I20260925 14:59:09.892360 13176 ts_tablet_manager.cc:1905] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:09.892673 13177 ts_tablet_manager.cc:1939] T e3b1e2f26adc4c7a97454c39c9657e77 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting consensus metadata
I20260925 14:59:09.894289 12628 catalog_manager.cc:4841] TS 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923): tablet e3b1e2f26adc4c7a97454c39c9657e77 (table test-table [id=ce67244c78714136832f9e5715eebe76]) successfully deleted
I20260925 14:59:09.900499 13154 tablet_bootstrap.cc:492] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: Bootstrap starting.
I20260925 14:59:09.902098 13154 tablet_bootstrap.cc:654] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:09.907209 13176 ts_tablet_manager.cc:1918] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:09.907406 13176 log.cc:1196] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/e3b1e2f26adc4c7a97454c39c9657e77
I20260925 14:59:09.907986 13176 ts_tablet_manager.cc:1939] T e3b1e2f26adc4c7a97454c39c9657e77 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting consensus metadata
I20260925 14:59:09.910000 12630 catalog_manager.cc:4841] TS 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061): tablet e3b1e2f26adc4c7a97454c39c9657e77 (table test-table [id=ce67244c78714136832f9e5715eebe76]) successfully deleted
I20260925 14:59:09.911368 13156 tablet_bootstrap.cc:492] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: No bootstrap required, opened a new log
I20260925 14:59:09.911592 13156 ts_tablet_manager.cc:1397] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.019s
I20260925 14:59:09.911734 13154 tablet_bootstrap.cc:492] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: No bootstrap required, opened a new log
I20260925 14:59:09.911878 13153 tablet_bootstrap.cc:492] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: No bootstrap required, opened a new log
I20260925 14:59:09.911998 13154 ts_tablet_manager.cc:1397] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260925 14:59:09.912045 13153 ts_tablet_manager.cc:1397] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: Time spent bootstrapping tablet: real 0.042s user 0.018s sys 0.023s
I20260925 14:59:09.912542 13156 raft_consensus.cc:348] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:09.912938 13156 raft_consensus.cc:374] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:09.912825 13154 raft_consensus.cc:348] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:09.912804 13153 raft_consensus.cc:348] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:09.913041 13154 raft_consensus.cc:374] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:09.913041 13153 raft_consensus.cc:374] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:09.913136 13153 raft_consensus.cc:729] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55b8b422e014257be586b950ebdc363, State: Initialized, Role: FOLLOWER
I20260925 14:59:09.913131 13156 raft_consensus.cc:729] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c569ea0a87b4bf0971de7f6aa60479a, State: Initialized, Role: FOLLOWER
I20260925 14:59:09.913452 13154 raft_consensus.cc:729] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Initialized, Role: FOLLOWER
I20260925 14:59:09.913410 13156 consensus_queue.cc:260] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:09.913480 13153 consensus_queue.cc:260] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:09.914137 13153 ts_tablet_manager.cc:1428] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 14:59:09.914499 13154 consensus_queue.cc:260] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:09.915103 13154 ts_tablet_manager.cc:1428] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:09.915467 13156 ts_tablet_manager.cc:1428] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 14:59:10.096002 13160 raft_consensus.cc:482] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:10.096294 13160 raft_consensus.cc:504] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.098218 13160 leader_election.cc:288] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061), a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121)
I20260925 14:59:10.113351 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.098956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:58303 (local address 127.4.243.194:42121)
0925 14:59:10.099156 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.099163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.107100 (+ 7937us) server_negotiation.cc:407] Connection header received
0925 14:59:10.107407 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.107415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.107502 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.107584 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:10.108478 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.109528 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.111379 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.112362 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.112511 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.112579 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.112945 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.113068 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:59:10.113544 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.098558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42121 (local address 127.4.243.195:58303)
0925 14:59:10.106989 (+ 8431us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:10.107010 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:10.107221 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:10.107775 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:10.107779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:10.107841 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:10.108216 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.108228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.109681 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.109689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.111222 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.111233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.112613 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.112621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.112771 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.112810 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 14:59:10.112915 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:59:10.113041 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8281,"mutex_wait_us":2,"thread_start_us":8111,"threads_started":1}
I20260925 14:59:10.114194 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93f1d556158410da28221ec24c2dcb0" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363" is_pre_election: true
I20260925 14:59:10.114471 12944 raft_consensus.cc:2454] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 0.
I20260925 14:59:10.114575 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.098686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:43061 (local address 127.4.243.195:53983)
0925 14:59:10.107404 (+ 8718us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:10.107419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:10.107821 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:10.108237 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:10.108243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:10.108272 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:10.108633 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.108641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.109718 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.109723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.111768 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.111787 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.113818 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.113829 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.114007 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.114043 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:59:10.114142 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:59:10.114267 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8556,"thread_start_us":8378,"threads_started":1}
I20260925 14:59:10.114982 13011 leader_election.cc:302] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c569ea0a87b4bf0971de7f6aa60479a, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:10.115125 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.099323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:53983 (local address 127.4.243.193:43061)
0925 14:59:10.099499 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.099506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.107518 (+ 8012us) server_negotiation.cc:407] Connection header received
0925 14:59:10.107994 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.108001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.108088 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.108198 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:10.108812 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.109549 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.111956 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.113031 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.113168 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.113252 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.114183 (+ 931us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.114325 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:59:10.115458 13160 raft_consensus.cc:2781] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:10.115574 13160 raft_consensus.cc:482] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:10.115674 13160 raft_consensus.cc:3037] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.115883 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93f1d556158410da28221ec24c2dcb0" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699b62e4b4c24853bddfa2d1126bfd21" is_pre_election: true
I20260925 14:59:10.116322 12805 raft_consensus.cc:2454] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 0.
I20260925 14:59:10.118328 13160 raft_consensus.cc:504] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.118970 13160 leader_election.cc:288] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 election: Requested vote from peers 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061), a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121)
I20260925 14:59:10.119340 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93f1d556158410da28221ec24c2dcb0" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699b62e4b4c24853bddfa2d1126bfd21"
I20260925 14:59:10.119432 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93f1d556158410da28221ec24c2dcb0" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363"
I20260925 14:59:10.119544 12805 raft_consensus.cc:3037] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.119649 12944 raft_consensus.cc:3037] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.121737 12805 raft_consensus.cc:2454] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 1.
I20260925 14:59:10.122131 13012 leader_election.cc:302] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c569ea0a87b4bf0971de7f6aa60479a, 699b62e4b4c24853bddfa2d1126bfd21; no voters:
I20260925 14:59:10.122409 13160 raft_consensus.cc:2781] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:10.122493 12944 raft_consensus.cc:2454] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 1.
I20260925 14:59:10.123127 13160 raft_consensus.cc:686] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 LEADER]: Becoming Leader. State: Replica: 1c569ea0a87b4bf0971de7f6aa60479a, State: Running, Role: LEADER
I20260925 14:59:10.123481 13160 consensus_queue.cc:237] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.127009 12643 catalog_manager.cc:5495] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195). New cstate: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } health_report { overall_health: HEALTHY } } }
Sep 25 14:59:10 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348346, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.254@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:10.200524 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.185008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37443 (local address 127.0.0.1:43136)
0925 14:59:10.185865 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:10.185885 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:10.186061 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:10.186511 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:10.186520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:10.187395 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:10.187647 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.187659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.188668 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.188675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.190256 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.190272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.191024 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.191034 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.191211 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.191925 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:10.191948 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:10.194668 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:10.197793 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:10.197806 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:10.197814 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:10.198288 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:10.198694 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:10.198701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:10.198704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:10.198851 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:10.199420 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:10.199428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:10.199596 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 14:59:10.199759 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 14:59:10.200039 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":577,"threads_started":1}
I20260925 14:59:10.200532 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.185059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.4.243.254:37443)
0925 14:59:10.185703 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.185709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.186034 (+ 325us) server_negotiation.cc:407] Connection header received
0925 14:59:10.186196 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.186241 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.186337 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.186419 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:10.187784 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.188530 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.190397 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.190850 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.190944 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.194852 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:10.194856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:10.194859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:10.194884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:10.197677 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:10.198410 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:10.198417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:10.198421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:10.198574 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:10.198949 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:10.198973 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:10.198976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:10.199298 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:10.199438 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.199836 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.200149 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":479,"thread_start_us":307,"threads_started":1}
I20260925 14:59:10.205433 12643 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:43136:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:59:10.206465 12643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:10.213147 12780 tablet_service.cc:1467] Processing CreateTablet for tablet 6318f14f4de347da99adc9747cee6613 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08d5812e4ad4496c8c294f9be558e6b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:59:10.213295 12919 tablet_service.cc:1467] Processing CreateTablet for tablet 6318f14f4de347da99adc9747cee6613 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08d5812e4ad4496c8c294f9be558e6b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:59:10.213557 13058 tablet_service.cc:1467] Processing CreateTablet for tablet 6318f14f4de347da99adc9747cee6613 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08d5812e4ad4496c8c294f9be558e6b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:59:10.213742 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6318f14f4de347da99adc9747cee6613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.213826 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6318f14f4de347da99adc9747cee6613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.214067 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6318f14f4de347da99adc9747cee6613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.220829 13154 tablet_bootstrap.cc:492] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: Bootstrap starting.
I20260925 14:59:10.220924 13156 tablet_bootstrap.cc:492] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: Bootstrap starting.
I20260925 14:59:10.222312 13156 tablet_bootstrap.cc:654] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.222611 13154 tablet_bootstrap.cc:654] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.223639 13153 tablet_bootstrap.cc:492] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: Bootstrap starting.
I20260925 14:59:10.224486 13156 tablet_bootstrap.cc:492] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: No bootstrap required, opened a new log
I20260925 14:59:10.224673 13156 ts_tablet_manager.cc:1397] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:59:10.225265 13153 tablet_bootstrap.cc:654] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.225235 13156 raft_consensus.cc:348] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.225466 13156 raft_consensus.cc:374] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.225566 13156 raft_consensus.cc:729] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c569ea0a87b4bf0971de7f6aa60479a, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.225795 13156 consensus_queue.cc:260] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.226181 13160 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.226437 13160 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:59:10.227439 13156 ts_tablet_manager.cc:1428] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:10.228888 13153 tablet_bootstrap.cc:492] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: No bootstrap required, opened a new log
I20260925 14:59:10.229039 13153 ts_tablet_manager.cc:1397] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260925 14:59:10.230182 13153 raft_consensus.cc:348] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.230409 13153 raft_consensus.cc:374] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.230491 13153 raft_consensus.cc:729] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55b8b422e014257be586b950ebdc363, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.230767 13153 consensus_queue.cc:260] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.231199 13159 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.231501 13159 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:59:10.232486 13153 ts_tablet_manager.cc:1428] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:59:10.236662 13154 tablet_bootstrap.cc:492] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: No bootstrap required, opened a new log
I20260925 14:59:10.236837 13154 ts_tablet_manager.cc:1397] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20260925 14:59:10.237579 13154 raft_consensus.cc:348] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.237830 13154 raft_consensus.cc:374] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.237910 13154 raft_consensus.cc:729] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.238137 13154 consensus_queue.cc:260] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.238828 13154 ts_tablet_manager.cc:1428] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:10.238651 13164 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.238989 13164 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:59:10.309257 13160 raft_consensus.cc:482] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:10.309458 13160 raft_consensus.cc:504] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.309993 13160 leader_election.cc:288] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061), a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121)
I20260925 14:59:10.310535 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318f14f4de347da99adc9747cee6613" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699b62e4b4c24853bddfa2d1126bfd21" is_pre_election: true
I20260925 14:59:10.310542 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318f14f4de347da99adc9747cee6613" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363" is_pre_election: true
I20260925 14:59:10.310839 12805 raft_consensus.cc:2454] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 0.
I20260925 14:59:10.310851 12944 raft_consensus.cc:2454] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 0.
I20260925 14:59:10.311266 13011 leader_election.cc:302] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c569ea0a87b4bf0971de7f6aa60479a, 699b62e4b4c24853bddfa2d1126bfd21, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:10.311576 13160 raft_consensus.cc:2781] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:10.311687 13160 raft_consensus.cc:482] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:10.311807 13160 raft_consensus.cc:3037] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.313840 13160 raft_consensus.cc:504] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.314332 13160 leader_election.cc:288] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 election: Requested vote from peers 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061), a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121)
I20260925 14:59:10.314770 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318f14f4de347da99adc9747cee6613" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363"
I20260925 14:59:10.314781 12805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6318f14f4de347da99adc9747cee6613" candidate_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699b62e4b4c24853bddfa2d1126bfd21"
I20260925 14:59:10.314962 12944 raft_consensus.cc:3037] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.315004 12805 raft_consensus.cc:3037] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.316804 12944 raft_consensus.cc:2454] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 1.
I20260925 14:59:10.316905 12805 raft_consensus.cc:2454] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c569ea0a87b4bf0971de7f6aa60479a in term 1.
I20260925 14:59:10.317255 13011 leader_election.cc:302] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c569ea0a87b4bf0971de7f6aa60479a, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:10.317523 13160 raft_consensus.cc:2781] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:10.317699 13160 raft_consensus.cc:686] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 LEADER]: Becoming Leader. State: Replica: 1c569ea0a87b4bf0971de7f6aa60479a, State: Running, Role: LEADER
I20260925 14:59:10.317947 13160 consensus_queue.cc:237] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.319366 13184 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: TxnStatusTablet state changed. Reason: New leader 1c569ea0a87b4bf0971de7f6aa60479a. Latest consensus state: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.319629 13184 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:59:10.320282 13197 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:59:10.320489 13197 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:59:10.320796 12643 catalog_manager.cc:5495] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195). New cstate: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } health_report { overall_health: HEALTHY } } }
I20260925 14:59:10.349068 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.342586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:43923 (local address 127.0.0.1:40454)
0925 14:59:10.342892 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:10.342915 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:10.343073 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:10.343694 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:10.343699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:10.343710 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:59:10.343932 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.343946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.345158 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.345167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.346462 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.346475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.347345 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.347351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.347530 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.347582 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:59:10.348492 (+ 910us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:59:10.348511 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 14:59:10.348588 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:59:10.348742 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 14:59:10.349118 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.342574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.4.243.195:43923)
0925 14:59:10.343328 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.343333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.343347 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:59:10.343414 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.343417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.343483 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.343680 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:10.344082 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.345033 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.346641 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.347121 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.347309 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.347807 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:10.348363 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:10.348540 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.348604 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.348684 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":432,"threads_started":1}
I20260925 14:59:10.353358 12944 raft_consensus.cc:1261] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Refusing update from remote peer 1c569ea0a87b4bf0971de7f6aa60479a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:10.353547 12805 raft_consensus.cc:1261] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Refusing update from remote peer 1c569ea0a87b4bf0971de7f6aa60479a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:10.354095 13184 consensus_queue.cc:1035] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:10.354485 13160 consensus_queue.cc:1035] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a [LEADER]: Connected to new peer: Peer: permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:10.359699 13159 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: TxnStatusTablet state changed. Reason: New leader 1c569ea0a87b4bf0971de7f6aa60479a. Latest consensus state: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.360052 13159 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:59:10.362010 13164 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: TxnStatusTablet state changed. Reason: New leader 1c569ea0a87b4bf0971de7f6aa60479a. Latest consensus state: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.362294 13164 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:59:10.364539 13184 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.364825 13184 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:59:10.365402 13159 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.365445 13184 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.365630 13159 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P a55b8b422e014257be586b950ebdc363: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:59:10.365677 13184 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P 1c569ea0a87b4bf0971de7f6aa60479a: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:59:10.367937 13164 tablet_replica.cc:439] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } } }
I20260925 14:59:10.368197 13164 tablet_replica.cc:442] T 6318f14f4de347da99adc9747cee6613 P 699b62e4b4c24853bddfa2d1126bfd21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:59:10.380792 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.375004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.4.243.195:43923)
0925 14:59:10.375217 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.375226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.375638 (+ 412us) server_negotiation.cc:407] Connection header received
0925 14:59:10.375812 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.375815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.375885 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.375957 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:10.376499 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.377253 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.379155 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.379580 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.379670 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.380124 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:10.380209 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:10.380272 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.380439 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.380498 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:59:10.396447 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.390825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:58763 (local address 127.4.243.195:43923)
0925 14:59:10.390990 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.390993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.391617 (+ 624us) server_negotiation.cc:407] Connection header received
0925 14:59:10.391794 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.391799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.391895 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.392000 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:10.392486 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.393251 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.394764 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.395197 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.395330 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.395710 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:10.395796 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:10.395914 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.396052 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.396177 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260925 14:59:10.396807 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.390966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:43923 (local address 127.4.243.195:58763)
0925 14:59:10.391525 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:10.391539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:10.391689 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:10.392025 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:10.392030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:10.392046 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:59:10.392359 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.392366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.393385 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.393393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.394585 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.394644 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.395381 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.395389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.395504 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.395548 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:59:10.395914 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:59:10.395924 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:59:10.396021 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:59:10.396279 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":74,"thread_start_us":294,"threads_started":1}
I20260925 14:59:10.405275 12805 raft_consensus.cc:1261] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Refusing update from remote peer 1c569ea0a87b4bf0971de7f6aa60479a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:10.405841 12944 raft_consensus.cc:1261] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Refusing update from remote peer 1c569ea0a87b4bf0971de7f6aa60479a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:10.405929 13160 consensus_queue.cc:1035] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [LEADER]: Connected to new peer: Peer: permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:10.406291 13184 consensus_queue.cc:1035] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:10.469265 12643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:10.469512 12643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:10.472360 12643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Sending DeleteTablet for 3 replicas of tablet a93f1d556158410da28221ec24c2dcb0
I20260925 14:59:10.473173 12780 tablet_service.cc:1514] Processing DeleteTablet for tablet a93f1d556158410da28221ec24c2dcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:10 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:42910
I20260925 14:59:10.473759 13218 tablet_replica.cc:330] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: stopping tablet replica
I20260925 14:59:10.474025 13218 raft_consensus.cc:2229] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:10.474239 13218 raft_consensus.cc:2258] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:10.475203 13218 ts_tablet_manager.cc:1905] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:10.478991 13058 tablet_service.cc:1514] Processing DeleteTablet for tablet a93f1d556158410da28221ec24c2dcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:10 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260925 14:59:10.479094 12919 tablet_service.cc:1514] Processing DeleteTablet for tablet a93f1d556158410da28221ec24c2dcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:10 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:41364
I20260925 14:59:10.496618 13218 ts_tablet_manager.cc:1918] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:59:10.496770 13218 log.cc:1196] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/a93f1d556158410da28221ec24c2dcb0
I20260925 14:59:10.497215 13218 ts_tablet_manager.cc:1939] T a93f1d556158410da28221ec24c2dcb0 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting consensus metadata
I20260925 14:59:10.498248 12630 catalog_manager.cc:4841] TS 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061): tablet a93f1d556158410da28221ec24c2dcb0 (table test-table [id=78bf4a13e7c64848a0d7e00593bd9db6]) successfully deleted
I20260925 14:59:10.498549 13220 tablet_replica.cc:330] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: stopping tablet replica
I20260925 14:59:10.498761 13220 raft_consensus.cc:2229] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:10.499009 13220 raft_consensus.cc:2258] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:10.500025 13220 ts_tablet_manager.cc:1905] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:10.507174 13220 ts_tablet_manager.cc:1918] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:59:10.507301 13220 log.cc:1196] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/a93f1d556158410da28221ec24c2dcb0
I20260925 14:59:10.507752 13220 ts_tablet_manager.cc:1939] T a93f1d556158410da28221ec24c2dcb0 P a55b8b422e014257be586b950ebdc363: Deleting consensus metadata
I20260925 14:59:10.508751 12629 catalog_manager.cc:4841] TS a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121): tablet a93f1d556158410da28221ec24c2dcb0 (table test-table [id=78bf4a13e7c64848a0d7e00593bd9db6]) successfully deleted
I20260925 14:59:10.517772 13219 tablet_replica.cc:330] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: stopping tablet replica
I20260925 14:59:10.517989 13219 raft_consensus.cc:2229] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:10.518664 13219 raft_consensus.cc:2258] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:10.519644 13219 ts_tablet_manager.cc:1905] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:10.527529 13219 ts_tablet_manager.cc:1918] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:59:10.527680 13219 log.cc:1196] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/a93f1d556158410da28221ec24c2dcb0
I20260925 14:59:10.528110 13219 ts_tablet_manager.cc:1939] T a93f1d556158410da28221ec24c2dcb0 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting consensus metadata
I20260925 14:59:10.529145 12628 catalog_manager.cc:4841] TS 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923): tablet a93f1d556158410da28221ec24c2dcb0 (table test-table [id=78bf4a13e7c64848a0d7e00593bd9db6]) successfully deleted
Sep 25 14:59:10 dist-test-slave-qxh0 krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:59:10 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:10 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:59:10.566126 13191 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:10.553559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.4.243.254:37443)
0925 14:59:10.553764 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.553771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.554272 (+ 501us) server_negotiation.cc:407] Connection header received
0925 14:59:10.554462 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.554468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.554548 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.554703 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:10.556428 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.557524 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.559377 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.560049 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.560179 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.565881 (+ 5702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43148: BlockingRecv error: recv got EOF from 127.0.0.1:43148 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 14:59:10 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM
I20260925 14:59:10.583168 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.568795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43158 (local address 127.4.243.254:37443)
0925 14:59:10.568982 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.568988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.569233 (+ 245us) server_negotiation.cc:407] Connection header received
0925 14:59:10.569373 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.569377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.569431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.569515 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:10.570915 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.571662 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.573496 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.573875 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.573975 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.578040 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:10.578045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:10.578048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:10.578079 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:10.580642 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:10.581276 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:10.581288 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:10.581291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:10.581428 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:10.581825 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:10.581830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:10.581833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:10.582083 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:10.582343 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.582613 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.582833 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:59:10.586582 12643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:10.587186 12643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:10.593860 12780 tablet_service.cc:1467] Processing CreateTablet for tablet c2dde6326447424e92f6bd42bc12f747 (DEFAULT_TABLE table=test-table [id=4c1cd6e360ab4bb0aed069630a9355dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:10.593901 12919 tablet_service.cc:1467] Processing CreateTablet for tablet c2dde6326447424e92f6bd42bc12f747 (DEFAULT_TABLE table=test-table [id=4c1cd6e360ab4bb0aed069630a9355dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:10.594269 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2dde6326447424e92f6bd42bc12f747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.594281 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2dde6326447424e92f6bd42bc12f747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.598869 13153 tablet_bootstrap.cc:492] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: Bootstrap starting.
I20260925 14:59:10.598819 13058 tablet_service.cc:1467] Processing CreateTablet for tablet c2dde6326447424e92f6bd42bc12f747 (DEFAULT_TABLE table=test-table [id=4c1cd6e360ab4bb0aed069630a9355dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:10.599221 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2dde6326447424e92f6bd42bc12f747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.600358 13154 tablet_bootstrap.cc:492] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: Bootstrap starting.
I20260925 14:59:10.600479 13153 tablet_bootstrap.cc:654] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.601877 13154 tablet_bootstrap.cc:654] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.609894 13156 tablet_bootstrap.cc:492] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: Bootstrap starting.
I20260925 14:59:10.611723 13156 tablet_bootstrap.cc:654] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.612134 13154 tablet_bootstrap.cc:492] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: No bootstrap required, opened a new log
I20260925 14:59:10.612279 13154 ts_tablet_manager.cc:1397] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20260925 14:59:10.612541 13153 tablet_bootstrap.cc:492] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: No bootstrap required, opened a new log
I20260925 14:59:10.612816 13153 ts_tablet_manager.cc:1397] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260925 14:59:10.613047 13154 raft_consensus.cc:348] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.613334 13154 raft_consensus.cc:374] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.613448 13154 raft_consensus.cc:729] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.613736 13154 consensus_queue.cc:260] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.614305 13153 raft_consensus.cc:348] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.614713 13153 raft_consensus.cc:374] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.615134 13153 raft_consensus.cc:729] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55b8b422e014257be586b950ebdc363, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.615468 13153 consensus_queue.cc:260] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.616255 13153 ts_tablet_manager.cc:1428] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:59:10.616869 13154 ts_tablet_manager.cc:1428] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 14:59:10.619012 13156 tablet_bootstrap.cc:492] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: No bootstrap required, opened a new log
I20260925 14:59:10.619215 13156 ts_tablet_manager.cc:1397] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 14:59:10.620064 13156 raft_consensus.cc:348] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.620280 13156 raft_consensus.cc:374] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.620380 13156 raft_consensus.cc:729] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c569ea0a87b4bf0971de7f6aa60479a, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.620640 13156 consensus_queue.cc:260] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.621145 13156 ts_tablet_manager.cc:1428] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 14:59:10.822499 13164 raft_consensus.cc:482] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:10.822767 13164 raft_consensus.cc:504] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.823407 13164 leader_election.cc:288] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121), 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923)
I20260925 14:59:10.823959 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2dde6326447424e92f6bd42bc12f747" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363" is_pre_election: true
I20260925 14:59:10.824008 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2dde6326447424e92f6bd42bc12f747" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" is_pre_election: true
I20260925 14:59:10.824213 12944 raft_consensus.cc:2454] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 0.
I20260925 14:59:10.824224 13078 raft_consensus.cc:2454] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 0.
I20260925 14:59:10.824687 12732 leader_election.cc:302] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699b62e4b4c24853bddfa2d1126bfd21, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:10.824985 13164 raft_consensus.cc:2781] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:10.825093 13164 raft_consensus.cc:482] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:10.825196 13164 raft_consensus.cc:3037] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.828135 13164 raft_consensus.cc:504] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.828752 13164 leader_election.cc:288] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 election: Requested vote from peers a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121), 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923)
I20260925 14:59:10.829175 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2dde6326447424e92f6bd42bc12f747" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363"
I20260925 14:59:10.829201 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2dde6326447424e92f6bd42bc12f747" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c569ea0a87b4bf0971de7f6aa60479a"
I20260925 14:59:10.829358 12944 raft_consensus.cc:3037] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.829416 13078 raft_consensus.cc:3037] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.831276 12944 raft_consensus.cc:2454] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 1.
I20260925 14:59:10.831692 12732 leader_election.cc:302] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699b62e4b4c24853bddfa2d1126bfd21, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:10.831921 13164 raft_consensus.cc:2781] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:10.832324 13164 raft_consensus.cc:686] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 LEADER]: Becoming Leader. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Running, Role: LEADER
I20260925 14:59:10.832465 13078 raft_consensus.cc:2454] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 1.
I20260925 14:59:10.832579 13164 consensus_queue.cc:237] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.835322 12641 catalog_manager.cc:5495] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "699b62e4b4c24853bddfa2d1126bfd21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:10.853621 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.848215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.4.243.193:43061)
0925 14:59:10.848732 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.848736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.848751 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:59:10.848798 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.848801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.848855 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.848939 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:10.849331 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.850066 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.852004 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.852385 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.852471 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:10.852930 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:10.853035 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:10.853096 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.853263 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.853318 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":305,"threads_started":1}
I20260925 14:59:10.856062 13078 raft_consensus.cc:1261] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Refusing update from remote peer 699b62e4b4c24853bddfa2d1126bfd21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:10.856120 12944 raft_consensus.cc:1261] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Refusing update from remote peer 699b62e4b4c24853bddfa2d1126bfd21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:10.856662 13236 consensus_queue.cc:1035] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:10.856999 13164 consensus_queue.cc:1035] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:10.869653 12640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:10.869957 12640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:10.873056 12640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Sending DeleteTablet for 3 replicas of tablet c2dde6326447424e92f6bd42bc12f747
I20260925 14:59:10.873837 12780 tablet_service.cc:1514] Processing DeleteTablet for tablet c2dde6326447424e92f6bd42bc12f747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:10 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:42910
I20260925 14:59:10.873960 12919 tablet_service.cc:1514] Processing DeleteTablet for tablet c2dde6326447424e92f6bd42bc12f747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:10 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:41364
I20260925 14:59:10.874078 13218 tablet_replica.cc:330] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: stopping tablet replica
I20260925 14:59:10.874193 13220 tablet_replica.cc:330] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: stopping tablet replica
I20260925 14:59:10.874253 13218 raft_consensus.cc:2229] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:10.874353 13220 raft_consensus.cc:2229] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:10.874519 13218 raft_consensus.cc:2258] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:10.874548 13220 raft_consensus.cc:2258] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:10.875612 13220 ts_tablet_manager.cc:1905] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:10.876247 13058 tablet_service.cc:1514] Processing DeleteTablet for tablet c2dde6326447424e92f6bd42bc12f747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:10 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260925 14:59:10.876513 13219 tablet_replica.cc:330] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: stopping tablet replica
I20260925 14:59:10.876685 13219 raft_consensus.cc:2229] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:10.876875 13219 raft_consensus.cc:2258] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:10.877938 13218 ts_tablet_manager.cc:1905] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:10.881851 13220 ts_tablet_manager.cc:1918] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:10.881976 13220 log.cc:1196] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/c2dde6326447424e92f6bd42bc12f747
I20260925 14:59:10.882362 13220 ts_tablet_manager.cc:1939] T c2dde6326447424e92f6bd42bc12f747 P a55b8b422e014257be586b950ebdc363: Deleting consensus metadata
I20260925 14:59:10.883441 12629 catalog_manager.cc:4841] TS a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121): tablet c2dde6326447424e92f6bd42bc12f747 (table test-table [id=4c1cd6e360ab4bb0aed069630a9355dd]) successfully deleted
I20260925 14:59:10.883878 12640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:10.884514 12640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:10.886267 13219 ts_tablet_manager.cc:1905] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:10.891693 13218 ts_tablet_manager.cc:1918] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:10.891886 13218 log.cc:1196] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/c2dde6326447424e92f6bd42bc12f747
I20260925 14:59:10.892311 13218 ts_tablet_manager.cc:1939] T c2dde6326447424e92f6bd42bc12f747 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting consensus metadata
I20260925 14:59:10.893363 12630 catalog_manager.cc:4841] TS 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061): tablet c2dde6326447424e92f6bd42bc12f747 (table test-table [id=4c1cd6e360ab4bb0aed069630a9355dd]) successfully deleted
I20260925 14:59:10.896466 13219 ts_tablet_manager.cc:1918] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:10.896494 12919 tablet_service.cc:1467] Processing CreateTablet for tablet 007801e7ac494ae09d269d1849d13357 (DEFAULT_TABLE table=test-table [id=4a669fdd56594f17afc8211710b3e410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:10.896898 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007801e7ac494ae09d269d1849d13357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.896893 13219 log.cc:1196] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/c2dde6326447424e92f6bd42bc12f747
I20260925 14:59:10.897377 13219 ts_tablet_manager.cc:1939] T c2dde6326447424e92f6bd42bc12f747 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting consensus metadata
I20260925 14:59:10.898407 12628 catalog_manager.cc:4841] TS 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923): tablet c2dde6326447424e92f6bd42bc12f747 (table test-table [id=4c1cd6e360ab4bb0aed069630a9355dd]) successfully deleted
I20260925 14:59:10.898886 13058 tablet_service.cc:1467] Processing CreateTablet for tablet 007801e7ac494ae09d269d1849d13357 (DEFAULT_TABLE table=test-table [id=4a669fdd56594f17afc8211710b3e410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:10.899260 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007801e7ac494ae09d269d1849d13357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.899765 12780 tablet_service.cc:1467] Processing CreateTablet for tablet 007801e7ac494ae09d269d1849d13357 (DEFAULT_TABLE table=test-table [id=4a669fdd56594f17afc8211710b3e410]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:10.900134 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007801e7ac494ae09d269d1849d13357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.904816 13153 tablet_bootstrap.cc:492] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: Bootstrap starting.
I20260925 14:59:10.905655 13156 tablet_bootstrap.cc:492] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: Bootstrap starting.
I20260925 14:59:10.906059 13154 tablet_bootstrap.cc:492] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: Bootstrap starting.
I20260925 14:59:10.906347 13153 tablet_bootstrap.cc:654] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.907284 13156 tablet_bootstrap.cc:654] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.907585 13154 tablet_bootstrap.cc:654] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.917186 13156 tablet_bootstrap.cc:492] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: No bootstrap required, opened a new log
I20260925 14:59:10.917209 13154 tablet_bootstrap.cc:492] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: No bootstrap required, opened a new log
I20260925 14:59:10.917399 13156 ts_tablet_manager.cc:1397] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260925 14:59:10.917918 13153 tablet_bootstrap.cc:492] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: No bootstrap required, opened a new log
I20260925 14:59:10.918087 13153 ts_tablet_manager.cc:1397] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.002s
I20260925 14:59:10.918081 13154 ts_tablet_manager.cc:1397] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20260925 14:59:10.918075 13156 raft_consensus.cc:348] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.918571 13156 raft_consensus.cc:374] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.918685 13156 raft_consensus.cc:729] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c569ea0a87b4bf0971de7f6aa60479a, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.918871 13153 raft_consensus.cc:348] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.919093 13153 raft_consensus.cc:374] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.918962 13156 consensus_queue.cc:260] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.919158 13153 raft_consensus.cc:729] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55b8b422e014257be586b950ebdc363, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.919373 13153 consensus_queue.cc:260] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.920107 13154 raft_consensus.cc:348] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.920380 13154 raft_consensus.cc:374] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.920514 13154 raft_consensus.cc:729] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.920821 13154 consensus_queue.cc:260] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.921500 13154 ts_tablet_manager.cc:1428] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260925 14:59:10.921793 13156 ts_tablet_manager.cc:1428] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 14:59:10.922317 13153 ts_tablet_manager.cc:1428] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:59:10.950023 13164 raft_consensus.cc:482] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:10.950232 13164 raft_consensus.cc:504] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.950870 13164 leader_election.cc:288] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121), 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923)
I20260925 14:59:10.951328 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007801e7ac494ae09d269d1849d13357" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363" is_pre_election: true
I20260925 14:59:10.951417 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007801e7ac494ae09d269d1849d13357" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" is_pre_election: true
I20260925 14:59:10.951579 12944 raft_consensus.cc:2454] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 0.
I20260925 14:59:10.951649 13078 raft_consensus.cc:2454] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 0.
I20260925 14:59:10.951970 12732 leader_election.cc:302] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699b62e4b4c24853bddfa2d1126bfd21, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:10.952245 13164 raft_consensus.cc:2781] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:10.952339 13164 raft_consensus.cc:482] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:10.952466 13164 raft_consensus.cc:3037] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.954382 13164 raft_consensus.cc:504] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.955391 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007801e7ac494ae09d269d1849d13357" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55b8b422e014257be586b950ebdc363"
I20260925 14:59:10.955454 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007801e7ac494ae09d269d1849d13357" candidate_uuid: "699b62e4b4c24853bddfa2d1126bfd21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c569ea0a87b4bf0971de7f6aa60479a"
I20260925 14:59:10.955578 12944 raft_consensus.cc:3037] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.955662 13078 raft_consensus.cc:3037] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.955152 13164 leader_election.cc:288] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 election: Requested vote from peers a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121), 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923)
I20260925 14:59:10.957974 12944 raft_consensus.cc:2454] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 1.
I20260925 14:59:10.958348 12732 leader_election.cc:302] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699b62e4b4c24853bddfa2d1126bfd21, a55b8b422e014257be586b950ebdc363; no voters:
I20260925 14:59:10.958643 13164 raft_consensus.cc:2781] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:10.958822 13164 raft_consensus.cc:686] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 LEADER]: Becoming Leader. State: Replica: 699b62e4b4c24853bddfa2d1126bfd21, State: Running, Role: LEADER
I20260925 14:59:10.959056 13078 raft_consensus.cc:2454] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699b62e4b4c24853bddfa2d1126bfd21 in term 1.
I20260925 14:59:10.959096 13164 consensus_queue.cc:237] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } }
I20260925 14:59:10.961550 12640 catalog_manager.cc:5495] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "699b62e4b4c24853bddfa2d1126bfd21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699b62e4b4c24853bddfa2d1126bfd21" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 43061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:11.001374 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.991428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:43923 (local address 127.4.243.193:58533)
0925 14:59:10.991998 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:10.992018 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:10.992140 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:10.995262 (+ 3122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:10.995267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:10.995281 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:59:10.995591 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.995601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.997027 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.997033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:10.998826 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:10.998840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.000034 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.000043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:11.000231 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.000311 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:59:11.000755 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:59:11.000767 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:59:11.000849 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:59:11.000982 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":375,"threads_started":1}
I20260925 14:59:11.001771 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.991716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:58533 (local address 127.4.243.195:43923)
0925 14:59:10.994964 (+ 3248us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.994970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.994988 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:59:10.995047 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.995052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.995124 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.995217 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:10.995741 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.996876 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:10.999021 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:10.999873 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.000059 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.000531 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:11.000651 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:11.000731 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:11.000878 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:59:11.000951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3144,"thread_start_us":3033,"threads_started":1}
I20260925 14:59:11.010252 13078 raft_consensus.cc:1261] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Refusing update from remote peer 699b62e4b4c24853bddfa2d1126bfd21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:11.010792 12944 raft_consensus.cc:1261] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Refusing update from remote peer 699b62e4b4c24853bddfa2d1126bfd21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:11.010881 13242 consensus_queue.cc:1035] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c569ea0a87b4bf0971de7f6aa60479a" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:11.011376 13164 consensus_queue.cc:1035] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55b8b422e014257be586b950ebdc363" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:11.042136 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:11.033258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:57579 (local address 127.4.243.193:43061)
0925 14:59:11.033455 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:11.033461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:11.036250 (+ 2789us) server_negotiation.cc:407] Connection header received
0925 14:59:11.036451 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:11.036456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:11.036531 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:11.036624 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:11.037160 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.038176 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.040322 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.040702 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.040791 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.041365 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:11.041485 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:11.041586 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:11.041794 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 14:59:11.041869 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 14:59:11.042409 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:11.033150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:43061 (local address 127.4.243.195:57579)
0925 14:59:11.036150 (+ 3000us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:11.036166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:11.036316 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:11.036636 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:11.036641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:11.036653 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:59:11.036977 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:11.036988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.038325 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.038330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:11.040173 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:11.040188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.040882 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.040894 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:11.041088 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.041138 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:59:11.041636 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:59:11.041676 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:59:11.041761 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:59:11.041909 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2846,"thread_start_us":2722,"threads_started":1}
I20260925 14:59:11.073417 12640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:11.073661 12640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:11.076454 12640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e863cf769e64c4fad86a1907d5b63c6: Sending DeleteTablet for 3 replicas of tablet 007801e7ac494ae09d269d1849d13357
I20260925 14:59:11.077296 12919 tablet_service.cc:1514] Processing DeleteTablet for tablet 007801e7ac494ae09d269d1849d13357 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:11 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:41364
I20260925 14:59:11.077442 12780 tablet_service.cc:1514] Processing DeleteTablet for tablet 007801e7ac494ae09d269d1849d13357 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:11 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:42910
I20260925 14:59:11.077606 13220 tablet_replica.cc:330] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: stopping tablet replica
I20260925 14:59:11.077701 13218 tablet_replica.cc:330] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: stopping tablet replica
I20260925 14:59:11.077777 13220 raft_consensus.cc:2229] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:11.077890 13218 raft_consensus.cc:2229] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:11.077994 13220 raft_consensus.cc:2258] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:11.078223 13218 raft_consensus.cc:2258] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:11.078447 13058 tablet_service.cc:1514] Processing DeleteTablet for tablet 007801e7ac494ae09d269d1849d13357 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:11 UTC) from {username='oryx', principal='oryx/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:40438
I20260925 14:59:11.078742 13219 tablet_replica.cc:330] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: stopping tablet replica
I20260925 14:59:11.078914 13219 raft_consensus.cc:2229] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:11.079116 13219 raft_consensus.cc:2258] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:11.079226 13218 ts_tablet_manager.cc:1905] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:11.080401 13219 ts_tablet_manager.cc:1905] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:11.083694 13220 ts_tablet_manager.cc:1905] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:11.083976 13218 ts_tablet_manager.cc:1918] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:59:11.084113 13218 log.cc:1196] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/007801e7ac494ae09d269d1849d13357
I20260925 14:59:11.084472 13218 ts_tablet_manager.cc:1939] T 007801e7ac494ae09d269d1849d13357 P 699b62e4b4c24853bddfa2d1126bfd21: Deleting consensus metadata
I20260925 14:59:11.085443 12630 catalog_manager.cc:4841] TS 699b62e4b4c24853bddfa2d1126bfd21 (127.4.243.193:43061): tablet 007801e7ac494ae09d269d1849d13357 (table test-table [id=4a669fdd56594f17afc8211710b3e410]) successfully deleted
I20260925 14:59:11.090034 13220 ts_tablet_manager.cc:1918] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:59:11.090157 13220 log.cc:1196] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/007801e7ac494ae09d269d1849d13357
I20260925 14:59:11.090518 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 12714
I20260925 14:59:11.090876 13220 ts_tablet_manager.cc:1939] T 007801e7ac494ae09d269d1849d13357 P a55b8b422e014257be586b950ebdc363: Deleting consensus metadata
I20260925 14:59:11.092204 12629 catalog_manager.cc:4841] TS a55b8b422e014257be586b950ebdc363 (127.4.243.194:42121): tablet 007801e7ac494ae09d269d1849d13357 (table test-table [id=4a669fdd56594f17afc8211710b3e410]) successfully deleted
I20260925 14:59:11.094986 13219 ts_tablet_manager.cc:1918] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:59:11.095147 13219 log.cc:1196] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/007801e7ac494ae09d269d1849d13357
I20260925 14:59:11.095638 13219 ts_tablet_manager.cc:1939] T 007801e7ac494ae09d269d1849d13357 P 1c569ea0a87b4bf0971de7f6aa60479a: Deleting consensus metadata
I20260925 14:59:11.096835 12628 catalog_manager.cc:4841] TS 1c569ea0a87b4bf0971de7f6aa60479a (127.4.243.195:43923): tablet 007801e7ac494ae09d269d1849d13357 (table test-table [id=4a669fdd56594f17afc8211710b3e410]) successfully deleted
Sep 25 14:59:11 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.193@KRBTEST.COM
I20260925 14:59:11.109511 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:11.091433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.4.243.193:43061)
0925 14:59:11.091614 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:11.091620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:11.092346 (+ 726us) server_negotiation.cc:407] Connection header received
0925 14:59:11.092620 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:11.092626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:11.092700 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:11.092769 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:11.094692 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.095762 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.097501 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.098194 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.098322 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.102716 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:11.102734 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:11.102755 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:11.102798 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:11.106670 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.107339 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.107349 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.107353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.107535 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.107883 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.107888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.107892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.108340 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:11.108643 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:11.108922 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:59:11.109133 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:59:11.110172 12845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:11.263682 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 12714
I20260925 14:59:11.285719 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 12854
Sep 25 14:59:11 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.194@KRBTEST.COM
I20260925 14:59:11.300149 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:11.286544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.4.243.194:42121)
0925 14:59:11.287079 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:11.287082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:11.287096 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:59:11.287146 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:11.287149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:11.287201 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:11.287317 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:11.288436 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.289167 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.290380 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.290959 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.291098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.294891 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:11.294903 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:11.294918 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:11.294953 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:11.297481 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.298078 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.298090 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.298092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.298207 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.298580 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.298586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.298589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.299007 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:11.299323 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:11.299591 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 14:59:11.299796 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":300,"threads_started":1}
I20260925 14:59:11.300810 12984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:11.393658 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 12854
I20260925 14:59:11.411240 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 12993
Sep 25 14:59:11 dist-test-slave-qxh0 krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.243.195@KRBTEST.COM
I20260925 14:59:11.425304 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:11.411966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.4.243.195:43923)
0925 14:59:11.412131 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:11.412136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:11.412159 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:59:11.412249 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:11.412253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:11.412310 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:11.412398 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:11.413584 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.414270 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.415483 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.415858 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.415957 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.420136 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:11.420152 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:11.420168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:11.420201 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:11.423129 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.423694 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.423701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.423704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.423845 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.424141 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.424145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.424146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.424481 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:11.424699 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:11.424905 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 14:59:11.425057 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 14:59:11.425890 13123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:11.568169 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 12993
I20260925 14:59:11.593535 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 12611
I20260925 14:59:11.594583 12683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:11.705466 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 12611
2026-09-25T14:59:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6510 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:11 dist-test-slave-qxh0 krb5kdc[13269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:11 dist-test-slave-qxh0 krb5kdc[13269](info): set up 2 sockets
Sep 25 14:59:11 dist-test-slave-qxh0 krb5kdc[13269](info): commencing operation
krb5kdc: starting...
W20260925 14:59:12.787611 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:59:12 dist-test-slave-qxh0 krb5kdc[13269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:59:12.990212 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:34363
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:34363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:13.295122 13282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:13.295367 13282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:13.295521 13282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:13.302325 13282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:13.302417 13282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:13.302546 13282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:13.302670 13282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:13.302723 13282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:13.311739 13282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:34363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:13.312767 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:13.313894 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:13.324317 13287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:13.324287 13290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:13.324450 13288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:13.325482 13282 server_base.cc:1034] running on GCE node
I20260925 14:59:13.326179 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:13.327420 13282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:13.328613 13282 hybrid_clock.cc:648] HybridClock initialized: now 1790348353328556 us; error 47 us; skew 500 ppm
I20260925 14:59:13.328943 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T14:59:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1689 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:13 dist-test-slave-qxh0 krb5kdc[13296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:13 dist-test-slave-qxh0 krb5kdc[13296](info): set up 2 sockets
Sep 25 14:59:13 dist-test-slave-qxh0 krb5kdc[13296](info): commencing operation
krb5kdc: starting...
W20260925 14:59:14.475569 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:59:14 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:59:14.680327 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44133
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:44133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:14.992865 13309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:14.993212 13309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:14.993379 13309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:15.000443 13309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:15.000581 13309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:15.000684 13309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:15.000782 13309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:15.000864 13309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:15.010501 13309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:44133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:15.011878 13309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:15.013170 13309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:15.024923 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:15.024876 13317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:15.025153 13314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:15.025996 13309 server_base.cc:1034] running on GCE node
I20260925 14:59:15.026824 13309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:15.028206 13309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:15.029404 13309 hybrid_clock.cc:648] HybridClock initialized: now 1790348355029355 us; error 41 us; skew 500 ppm
I20260925 14:59:15.029748 13309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:15 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348355, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:15.036780 13309 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:59:15.038333 13309 webserver.cc:460] Webserver started at http://127.4.243.254:40159/ using document root <none> and password file <none>
I20260925 14:59:15.039105 13309 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:15.039244 13309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:15.039551 13309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:15.042426 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "fc737cf811ec47b2a162b2eb231dc2a2"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "52699d619e96dfc47942dedc9a124057"
server_key_iv: "0aec97445a7ed139d23ccd1299950f67"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.043257 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "fc737cf811ec47b2a162b2eb231dc2a2"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "52699d619e96dfc47942dedc9a124057"
server_key_iv: "0aec97445a7ed139d23ccd1299950f67"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.048079 13309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:59:15.051246 13324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:15.052517 13309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:15.052692 13309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "fc737cf811ec47b2a162b2eb231dc2a2"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "52699d619e96dfc47942dedc9a124057"
server_key_iv: "0aec97445a7ed139d23ccd1299950f67"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.052872 13309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:15.065945 13309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:15.070200 13309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:15.070478 13309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:15.087898 13309 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:44133
I20260925 14:59:15.087905 13375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:44133 every 8 connection(s)
I20260925 14:59:15.089215 13309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:15.092772 13376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:15.098918 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 13309
I20260925 14:59:15.099169 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:59:15.099947 5071 external_mini_cluster.cc:1428] Setting key 7843b74bb4bcf5ee5368f4f6b0386a7d
I20260925 14:59:15.103711 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Bootstrap starting.
I20260925 14:59:15.107347 13376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:15.108376 13376 log.cc:824] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:15.111378 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: No bootstrap required, opened a new log
Sep 25 14:59:15 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:15.119745 13376 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:15.120102 13376 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:15.120229 13376 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc737cf811ec47b2a162b2eb231dc2a2, State: Initialized, Role: FOLLOWER
I20260925 14:59:15.120849 13376 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:15.121106 13376 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:15.121237 13376 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:15.121434 13376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:15.122713 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:15.101965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.4.243.254:44133)
0925 14:59:15.102759 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:15.102770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:15.102809 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:59:15.104577 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:15.104599 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:15.104993 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:15.105303 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:15.107094 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.108534 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.110130 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.110839 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.110974 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:15.115319 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:15.115327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:15.115339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:15.115378 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:15.118304 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.119083 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.119091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.119106 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.119263 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.119639 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.119648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.119650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.120260 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:15.120559 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:15.121011 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 14:59:15.121637 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":360,"threads_started":1}
I20260925 14:59:15.123858 13376 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:15.124354 13376 leader_election.cc:302] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc737cf811ec47b2a162b2eb231dc2a2; no voters:
I20260925 14:59:15.125010 13376 leader_election.cc:288] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:15.125119 13381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:15.128652 13376 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:15.129047 13381 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 LEADER]: Becoming Leader. State: Replica: fc737cf811ec47b2a162b2eb231dc2a2, State: Running, Role: LEADER
I20260925 14:59:15.129581 13381 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:15.132979 13383 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc737cf811ec47b2a162b2eb231dc2a2. Latest consensus state: current_term: 1 leader_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } } }
I20260925 14:59:15.133049 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } } }
I20260925 14:59:15.133332 13383 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:15.133401 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:15.136272 13391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:15.140496 13391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:15.148785 13391 catalog_manager.cc:1347] Generated new cluster ID: fcdb4dc628534458a355ff74ecb0abb1
I20260925 14:59:15.148912 13391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:15.189237 13391 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:15.191206 13391 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:15.205037 13391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Generated new TSK 0
I20260925 14:59:15.205853 13391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:59:15.272009 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44133
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:15.597469 13402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:15.597733 13402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:15.597899 13402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:15.605077 13402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:15.605201 13402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:15.605384 13402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:59:15.614653 13402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:15.615864 13402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:15.617153 13402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:15.630494 13408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:15.630520 13407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:15.630556 13410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:15.631325 13402 server_base.cc:1034] running on GCE node
I20260925 14:59:15.632005 13402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:15.632984 13402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:15.634212 13402 hybrid_clock.cc:648] HybridClock initialized: now 1790348355634152 us; error 52 us; skew 500 ppm
I20260925 14:59:15.634528 13402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:15 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348355, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:15.637439 13402 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:59:15.638787 13402 webserver.cc:460] Webserver started at http://127.4.243.193:34543/ using document root <none> and password file <none>
I20260925 14:59:15.639432 13402 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:15.639523 13402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:15.639755 13402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:15.642433 13402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "2af9ac6723894828be9c58b6b054a52f"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "8266634e29fe13251f789cf1939d1489"
server_key_iv: "649e5cf89581e2ba921232c870e950af"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.643285 13402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "2af9ac6723894828be9c58b6b054a52f"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "8266634e29fe13251f789cf1939d1489"
server_key_iv: "649e5cf89581e2ba921232c870e950af"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.647976 13402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:59:15.651050 13417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:15.652321 13402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 14:59:15.652496 13402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "2af9ac6723894828be9c58b6b054a52f"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "8266634e29fe13251f789cf1939d1489"
server_key_iv: "649e5cf89581e2ba921232c870e950af"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.652668 13402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:15.676165 13402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:15.680127 13402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:15.680413 13402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:15.681361 13402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:15.682696 13402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:15.682765 13402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:15.682914 13402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:15.682964 13402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:15.715524 13402 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:36289
I20260925 14:59:15.715626 13529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:36289 every 8 connection(s)
I20260925 14:59:15.716745 13402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:15.724777 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 13402
I20260925 14:59:15.725009 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:15.725775 5071 external_mini_cluster.cc:1428] Setting key a84c496403d4390f3552b6dbb9b73ea3
Sep 25 14:59:15 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348355, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:15.737119 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:15.719173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:52269 (local address 127.4.243.254:44133)
0925 14:59:15.719580 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:15.719584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:15.720586 (+ 1002us) server_negotiation.cc:407] Connection header received
0925 14:59:15.722250 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:15.722255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:15.722317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:15.722428 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:15.723820 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.724521 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.726766 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.727226 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.727321 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:15.731295 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:15.731304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:15.731312 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:15.731353 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:15.733983 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.734683 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.734690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.734695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.734839 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.735236 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.735243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.735249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.735606 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:15.735749 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:15.736351 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 14:59:15.736621 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260925 14:59:15.737741 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:15.719745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:52269)
0925 14:59:15.720432 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:15.720467 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:15.722003 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:15.722664 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:15.722672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:15.723147 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:15.723668 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:15.723683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.724698 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.724707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:15.726643 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:15.726662 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.727454 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.727461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:15.727713 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:15.728360 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:15.728380 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:15.731047 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:15.734130 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:15.734136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:15.734147 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:15.734532 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:15.734956 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:15.734959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:15.734961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:15.735117 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:15.735725 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:15.735731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:15.735998 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 14:59:15.736198 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 14:59:15.736577 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":269,"threads_started":1}
I20260925 14:59:15.739295 13530 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44133
I20260925 14:59:15.739616 13530 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:15.740451 13530 heartbeater.cc:511] Master 127.4.243.254:44133 requested a full tablet report, sending...
I20260925 14:59:15.742969 13341 ts_manager.cc:194] Registered new tserver with Master: 2af9ac6723894828be9c58b6b054a52f (127.4.243.193:36289)
I20260925 14:59:15.745226 13341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:52269
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:59:15.773761 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44133
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:16.097282 13536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:16.097654 13536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:16.097836 13536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:16.109243 13536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:16.109377 13536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:16.109563 13536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:59:16.119765 13536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:16.121032 13536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:16.122351 13536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:16.135281 13541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:16.135234 13542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:16.135198 13544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:16.136039 13536 server_base.cc:1034] running on GCE node
I20260925 14:59:16.136615 13536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:16.137504 13536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:16.138688 13536 hybrid_clock.cc:648] HybridClock initialized: now 1790348356138649 us; error 33 us; skew 500 ppm
I20260925 14:59:16.138994 13536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:16 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:16.142144 13536 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:59:16.143612 13536 webserver.cc:460] Webserver started at http://127.4.243.194:40745/ using document root <none> and password file <none>
I20260925 14:59:16.144258 13536 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:16.144354 13536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:16.144610 13536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:16.147455 13536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "844fae1db2844091b06c5e6162dcc44e"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "a656ccc812a35e830ac63f7295de178d"
server_key_iv: "035a4a71688d2629ea82ccbecbea4cf1"
server_key_version: "encryptionkey@0"
I20260925 14:59:16.148152 13536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "844fae1db2844091b06c5e6162dcc44e"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "a656ccc812a35e830ac63f7295de178d"
server_key_iv: "035a4a71688d2629ea82ccbecbea4cf1"
server_key_version: "encryptionkey@0"
I20260925 14:59:16.152921 13536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:59:16.156289 13551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:16.157639 13536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:59:16.157756 13536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "844fae1db2844091b06c5e6162dcc44e"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "a656ccc812a35e830ac63f7295de178d"
server_key_iv: "035a4a71688d2629ea82ccbecbea4cf1"
server_key_version: "encryptionkey@0"
I20260925 14:59:16.157902 13536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:16.171269 13536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:16.175019 13536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:16.175249 13536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:16.176091 13536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:16.177305 13536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:16.177382 13536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:16.177469 13536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:16.177516 13536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:16.210151 13536 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:41851
I20260925 14:59:16.210196 13663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:41851 every 8 connection(s)
I20260925 14:59:16.211468 13536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:16.212915 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 13536
I20260925 14:59:16.213115 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:16.213603 5071 external_mini_cluster.cc:1428] Setting key 8c7ce6e2388974a920ec1558bff43da7
Sep 25 14:59:16 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:16.233707 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:16.214042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:35875 (local address 127.4.243.254:44133)
0925 14:59:16.214238 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:16.214244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:16.215888 (+ 1644us) server_negotiation.cc:407] Connection header received
0925 14:59:16.217684 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:16.217689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:16.217770 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:16.217893 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:16.219525 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.220232 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.221962 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.222658 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.222792 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:16.226393 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:16.226408 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:16.226416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:16.226459 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:16.230167 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.231105 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.231114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.231120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.231307 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.231776 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.231781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.231785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.232197 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:16.232363 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:16.233071 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 14:59:16.233316 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:59:16.234319 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:16.214715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:35875)
0925 14:59:16.215748 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:16.215783 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:16.217432 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:16.218162 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:16.218173 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:16.218780 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:16.219355 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:16.219372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.220384 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.220389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:16.221814 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:16.221826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.222881 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.222887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:16.223090 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:16.223777 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:16.223799 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:16.226150 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:16.230404 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:16.230411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:16.230429 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:16.230914 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:16.231430 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:16.231433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:16.231435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:16.231650 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:16.232344 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:16.232349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:16.232703 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 14:59:16.232914 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 14:59:16.233271 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":564,"threads_started":1}
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 14:59:16.235813 13664 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44133
I20260925 14:59:16.236090 13664 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:16.236799 13664 heartbeater.cc:511] Master 127.4.243.254:44133 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 14:59:16.238234 13341 ts_manager.cc:194] Registered new tserver with Master: 844fae1db2844091b06c5e6162dcc44e (127.4.243.194:41851)
I20260925 14:59:16.239101 13341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:35875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:59:16.264225 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44133
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:16.581372 13669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:16.581626 13669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:16.581784 13669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:16.588799 13669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:16.588901 13669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:16.589013 13669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:59:16.598233 13669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:16.599422 13669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:16.600621 13669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:16.613098 13674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:16.613152 13677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:16.613171 13675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:16.614181 13669 server_base.cc:1034] running on GCE node
I20260925 14:59:16.614778 13669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:16.615576 13669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:16.616756 13669 hybrid_clock.cc:648] HybridClock initialized: now 1790348356616720 us; error 31 us; skew 500 ppm
I20260925 14:59:16.617059 13669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:16 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:16.619989 13669 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:59:16.621333 13669 webserver.cc:460] Webserver started at http://127.4.243.195:36841/ using document root <none> and password file <none>
I20260925 14:59:16.621932 13669 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:16.622026 13669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:16.622388 13669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:16.625239 13669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "7fc3603b7cd24a18a307ac575f8d408f"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "458ff8c04ba80f1404e4b90e07dc8c06"
server_key_iv: "72d6e67a3f0988c9ca6aaefa9f4f4107"
server_key_version: "encryptionkey@0"
I20260925 14:59:16.625975 13669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "7fc3603b7cd24a18a307ac575f8d408f"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "458ff8c04ba80f1404e4b90e07dc8c06"
server_key_iv: "72d6e67a3f0988c9ca6aaefa9f4f4107"
server_key_version: "encryptionkey@0"
I20260925 14:59:16.630887 13669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 14:59:16.633957 13684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:16.635195 13669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:16.635345 13669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "7fc3603b7cd24a18a307ac575f8d408f"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "458ff8c04ba80f1404e4b90e07dc8c06"
server_key_iv: "72d6e67a3f0988c9ca6aaefa9f4f4107"
server_key_version: "encryptionkey@0"
I20260925 14:59:16.635525 13669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:16.663609 13669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:16.667397 13669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:16.667670 13669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:16.668573 13669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:16.669804 13669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:16.669903 13669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:16.670046 13669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:16.670125 13669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:16.701162 13669 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42435
I20260925 14:59:16.701210 13796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42435 every 8 connection(s)
I20260925 14:59:16.702440 13669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:16.703825 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 13669
I20260925 14:59:16.704070 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:16.704574 5071 external_mini_cluster.cc:1428] Setting key 6fa5d2ea6182253e2ece93242df6a62c
Sep 25 14:59:16 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:16.722043 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:16.705341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:55457 (local address 127.4.243.254:44133)
0925 14:59:16.705528 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:16.705534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:16.706978 (+ 1444us) server_negotiation.cc:407] Connection header received
0925 14:59:16.708740 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:16.708745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:16.708824 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:16.708928 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:16.710310 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.711109 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.712669 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.713190 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.713316 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:16.716533 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:16.716538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:16.716542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:16.716573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:16.719577 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.720244 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.720249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.720253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.720390 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.720733 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.720736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.720738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.721007 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:16.721111 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:16.721656 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 14:59:16.721798 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:59:16.722927 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:16.706008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:55457)
0925 14:59:16.706855 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:16.706891 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:16.708506 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:16.709162 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:16.709170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:16.709637 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:16.710162 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:16.710179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.711249 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.711254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:16.712520 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:16.712531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.713396 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.713405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:16.713647 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:16.714227 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:16.714247 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:16.716347 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:16.719720 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:16.719729 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:16.719743 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:16.720131 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:16.720482 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:16.720485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:16.720487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:16.720649 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:16.721108 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:16.721112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:16.721378 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 14:59:16.721585 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 14:59:16.721904 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":359,"threads_started":1}
I20260925 14:59:16.724226 13797 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44133
I20260925 14:59:16.724503 13797 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:16.725159 13797 heartbeater.cc:511] Master 127.4.243.254:44133 requested a full tablet report, sending...
I20260925 14:59:16.726397 13341 ts_manager.cc:194] Registered new tserver with Master: 7fc3603b7cd24a18a307ac575f8d408f (127.4.243.195:42435)
I20260925 14:59:16.727173 13341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:55457
I20260925 14:59:16.731606 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:16.749056 13530 heartbeater.cc:503] Master 127.4.243.254:44133 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab.
Entry for principal oryx/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab.
WARNING: no policy specified for oryx/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab.
Entry for principal oryx/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab.
WARNING: no policy specified for oryx/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab.
Entry for principal oryx/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab.
I20260925 14:59:16.861102 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 13402
Sep 25 14:59:16 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:59:16.877293 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:16.862262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.4.243.193:36289)
0925 14:59:16.863023 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:16.863034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:16.863077 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:59:16.863204 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:16.863212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:16.863430 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:16.863614 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:16.865336 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.866549 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.867966 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.868502 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.868604 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:16.872350 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:16.872360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:16.872374 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:16.872405 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:16.874992 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.875551 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.875558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.875560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.875676 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.876003 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.876006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.876008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.876301 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:16.876504 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:16.876722 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:59:16.876898 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":428,"threads_started":1}
I20260925 14:59:16.878499 13525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:16.959894 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 13402
I20260925 14:59:16.976465 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 13536
Sep 25 14:59:16 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:59:16.991997 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:16.977381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.4.243.194:41851)
0925 14:59:16.977985 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:16.977992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:16.978017 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:59:16.978074 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:16.978080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:16.978305 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:16.978441 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:16.979637 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.980619 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.982023 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:16.982565 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:16.982730 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:16.986806 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:16.986817 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:16.986831 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:16.986862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:16.989569 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.990158 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.990164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.990166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.990278 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:16.990587 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:16.990621 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:16.990624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:16.990927 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:16.991145 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:16.991423 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:59:16.991614 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":314,"threads_started":1}
I20260925 14:59:16.993175 13659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:17.075008 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 13536
I20260925 14:59:17.090576 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 13669
Sep 25 14:59:17 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:59:17.106945 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:17.091494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.4.243.195:42435)
0925 14:59:17.092129 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:17.092136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:17.092164 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:59:17.092221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:17.092227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:17.092434 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:17.092568 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:17.094136 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:17.095330 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:17.096591 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:17.097174 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:17.097276 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:17.101649 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:17.101661 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:17.101675 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:17.101705 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:17.104468 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:17.105151 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:17.105158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:17.105161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:17.105330 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:17.105653 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:17.105657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:17.105659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:17.105968 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:17.106222 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:17.106430 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 14:59:17.106688 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":301,"threads_started":1}
I20260925 14:59:17.108219 13792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:17.193367 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 13669
I20260925 14:59:17.208911 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 13309
I20260925 14:59:17.210078 13371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:17.310084 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 13309
I20260925 14:59:17.330418 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44133
--webserver_interface=127.4.243.254
--webserver_port=40159
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:44133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:17.644802 13814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:17.645092 13814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:17.645247 13814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:17.652307 13814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:17.652397 13814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:17.652503 13814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:17.652614 13814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:17.652675 13814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:17.661867 13814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:44133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=40159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:17.663125 13814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:17.664427 13814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:17.675874 13822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:17.675825 13819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:17.675951 13820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:17.676661 13814 server_base.cc:1034] running on GCE node
I20260925 14:59:17.677376 13814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:17.678797 13814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:17.680038 13814 hybrid_clock.cc:648] HybridClock initialized: now 1790348357679975 us; error 55 us; skew 500 ppm
I20260925 14:59:17.680405 13814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:17 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348357, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:17.683355 13814 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:59:17.684633 13814 webserver.cc:460] Webserver started at http://127.4.243.254:40159/ using document root <none> and password file <none>
I20260925 14:59:17.685271 13814 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:17.685376 13814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:17.690296 13814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:17.692765 13829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:17.693980 13814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:59:17.694154 13814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "fc737cf811ec47b2a162b2eb231dc2a2"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "52699d619e96dfc47942dedc9a124057"
server_key_iv: "0aec97445a7ed139d23ccd1299950f67"
server_key_version: "encryptionkey@0"
I20260925 14:59:17.695168 13814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:17.721920 13814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:17.725723 13814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:17.725991 13814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:17.743826 13814 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:44133
I20260925 14:59:17.743849 13880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:44133 every 8 connection(s)
I20260925 14:59:17.745328 13814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:17.751075 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 13814
I20260925 14:59:17.752044 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:36289
--local_ip_for_outbound_sockets=127.4.243.193
--tserver_master_addrs=127.4.243.254:44133
--webserver_port=34543
--webserver_interface=127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:59:17.752516 13881 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:59:17.757321 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Bootstrap starting.
I20260925 14:59:17.776221 13881 log.cc:824] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:17.784897 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:59:17.785450 13881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Bootstrap complete.
I20260925 14:59:17.794162 13881 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:17.795104 13881 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc737cf811ec47b2a162b2eb231dc2a2, State: Initialized, Role: FOLLOWER
I20260925 14:59:17.795756 13881 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:17.796003 13881 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:17.796126 13881 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:17.796324 13881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:59:17.800767 13881 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:17.801235 13881 leader_election.cc:302] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc737cf811ec47b2a162b2eb231dc2a2; no voters:
I20260925 14:59:17.801898 13881 leader_election.cc:288] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:59:17.802032 13885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:59:17.804284 13881 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:17.809243 13885 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [term 2 LEADER]: Becoming Leader. State: Replica: fc737cf811ec47b2a162b2eb231dc2a2, State: Running, Role: LEADER
I20260925 14:59:17.809870 13885 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } }
I20260925 14:59:17.815205 13886 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } } }
I20260925 14:59:17.815552 13886 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:17.820355 13887 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc737cf811ec47b2a162b2eb231dc2a2. Latest consensus state: current_term: 2 leader_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc737cf811ec47b2a162b2eb231dc2a2" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44133 } } }
I20260925 14:59:17.821267 13887 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:17.832211 13899 catalog_manager.cc:1259] Loaded cluster ID: fcdb4dc628534458a355ff74ecb0abb1
I20260925 14:59:17.832338 13899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: loading cluster ID for follower catalog manager: success
I20260925 14:59:17.836256 13899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: acquiring CA information for follower catalog manager: success
I20260925 14:59:17.838855 13899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:59:17.839056 13900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:17.840821 13900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:17.841678 13900 catalog_manager.cc:1259] Loaded cluster ID: fcdb4dc628534458a355ff74ecb0abb1
I20260925 14:59:17.841799 13900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:17.842998 13900 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:17.843834 13900 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc737cf811ec47b2a162b2eb231dc2a2: Loaded TSK: 0
I20260925 14:59:17.844550 13900 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:59:18.144351 13883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:18.144686 13883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:18.144847 13883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:18.151990 13883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:18.152148 13883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:18.152318 13883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:59:18.161645 13883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.193.keytab
--principal=oryx/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:36289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=34543
--tserver_master_addrs=127.4.243.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:18.162979 13883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:18.164219 13883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:18.178754 13908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:18.178885 13907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:18.179785 13910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:18.179905 13883 server_base.cc:1034] running on GCE node
I20260925 14:59:18.180550 13883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:18.181545 13883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:18.182778 13883 hybrid_clock.cc:648] HybridClock initialized: now 1790348358182724 us; error 55 us; skew 500 ppm
I20260925 14:59:18.183139 13883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:18 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348358, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:18.186439 13883 init.cc:377] Logged in from keytab as oryx/127.4.243.193@KRBTEST.COM (short username oryx)
I20260925 14:59:18.187999 13883 webserver.cc:460] Webserver started at http://127.4.243.193:34543/ using document root <none> and password file <none>
I20260925 14:59:18.188683 13883 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:18.188818 13883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:18.194285 13883 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 14:59:18.196971 13917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:18.198364 13883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 14:59:18.198529 13883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "2af9ac6723894828be9c58b6b054a52f"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-qxh0"
server_key: "8266634e29fe13251f789cf1939d1489"
server_key_iv: "649e5cf89581e2ba921232c870e950af"
server_key_version: "encryptionkey@0"
I20260925 14:59:18.199574 13883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:18.236758 13883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:18.241123 13883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:18.241402 13883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:18.242373 13883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:18.243822 13883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:18.243947 13883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:18.244078 13883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:18.244184 13883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:18.277426 13883 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:36289
I20260925 14:59:18.277519 14029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:36289 every 8 connection(s)
I20260925 14:59:18.278888 13883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:18.286660 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 13883
I20260925 14:59:18.287235 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:41851
--local_ip_for_outbound_sockets=127.4.243.194
--tserver_master_addrs=127.4.243.254:44133
--webserver_port=40745
--webserver_interface=127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:59:18 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:18 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:59:18.611440 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:18.611737 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:18.611909 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:18.619158 14034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:18.619256 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:18.619467 14034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:59:18.628906 14034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.194.keytab
--principal=oryx/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:41851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=40745
--tserver_master_addrs=127.4.243.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:18.630187 14034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:18.631472 14034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:18.648530 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:18.649197 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:18.649659 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:18.649698 14034 server_base.cc:1034] running on GCE node
I20260925 14:59:18.650681 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:18.652153 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:18.653445 14034 hybrid_clock.cc:648] HybridClock initialized: now 1790348358653366 us; error 48 us; skew 500 ppm
I20260925 14:59:18.653945 14034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:18 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348358, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:18.665693 14034 init.cc:377] Logged in from keytab as oryx/127.4.243.194@KRBTEST.COM (short username oryx)
I20260925 14:59:18.668081 14034 webserver.cc:460] Webserver started at http://127.4.243.194:40745/ using document root <none> and password file <none>
I20260925 14:59:18.669313 14034 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:18.669517 14034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:18.679064 14034 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 14:59:18.683300 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:18.685366 14034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 14:59:18.685611 14034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "844fae1db2844091b06c5e6162dcc44e"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "a656ccc812a35e830ac63f7295de178d"
server_key_iv: "035a4a71688d2629ea82ccbecbea4cf1"
server_key_version: "encryptionkey@0"
I20260925 14:59:18.687526 14034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:18.726374 14034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:18.732594 14034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:18.732877 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:18.733911 14034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:18.735318 14034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:18.735457 14034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:18.735589 14034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:18.735683 14034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:18.777643 14162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:41851 every 8 connection(s)
I20260925 14:59:18.777652 14034 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:41851
I20260925 14:59:18.780346 14034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:18.790663 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14034
I20260925 14:59:18.791553 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:42435
--local_ip_for_outbound_sockets=127.4.243.195
--tserver_master_addrs=127.4.243.254:44133
--webserver_port=36841
--webserver_interface=127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:38691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:59:18 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:18 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:59:19.443760 14167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:19.444154 14167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:19.444303 14167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:19.456076 14167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:19.456208 14167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:19.456377 14167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:59:19.472749 14167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/krb5kdc/oryx_127.4.243.195.keytab
--principal=oryx/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:42435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=36841
--tserver_master_addrs=127.4.243.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:19.474479 14167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:19.476356 14167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:19.497014 14173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:19.498533 14174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:19.505650 14176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:19.512413 14167 server_base.cc:1034] running on GCE node
I20260925 14:59:19.513185 14167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:19.514420 14167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:19.517803 14167 hybrid_clock.cc:648] HybridClock initialized: now 1790348359517719 us; error 89 us; skew 500 ppm
I20260925 14:59:19.518201 14167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:19 dist-test-slave-qxh0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348359, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:19.523240 14167 init.cc:377] Logged in from keytab as oryx/127.4.243.195@KRBTEST.COM (short username oryx)
I20260925 14:59:19.525465 14167 webserver.cc:460] Webserver started at http://127.4.243.195:36841/ using document root <none> and password file <none>
I20260925 14:59:19.526373 14167 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:19.526510 14167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:19.533901 14167 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 14:59:19.537392 14183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:19.539260 14167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 14:59:19.539412 14167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "7fc3603b7cd24a18a307ac575f8d408f"
format_stamp: "Formatted at 2026-09-25 14:59:16 on dist-test-slave-qxh0"
server_key: "458ff8c04ba80f1404e4b90e07dc8c06"
server_key_iv: "72d6e67a3f0988c9ca6aaefa9f4f4107"
server_key_version: "encryptionkey@0"
I20260925 14:59:19.540644 14167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260925 14:59:19.555549 14033 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.263s user 0.004s sys 0.011s
W20260925 14:59:19.557919 14032 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:18.281684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:56797 (local address 127.4.243.254:44133)
0925 14:59:18.282375 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:18.282390 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:18.282687 (+ 297us) server_negotiation.cc:407] Connection header received
0925 14:59:18.284691 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:18.284703 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:18.285024 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:18.285349 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:18.286894 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.288138 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.290493 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.291165 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.291289 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.557083 (+1265794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:56797: BlockingRecv error: recv got EOF from 127.4.243.193:56797 (error 108)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":242,"threads_started":1}
I20260925 14:59:19.558211 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:18.281737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:56797)
0925 14:59:18.282499 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:18.282534 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:18.284219 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:18.285428 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:18.285435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:18.285894 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:18.286669 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:18.286694 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.288304 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.288312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:18.290345 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:18.290361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.291391 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.291400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:18.291692 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:18.292411 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:18.292458 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:19.556756 (+1264298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":358,"threads_started":1}
W20260925 14:59:19.559188 14030 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:44133 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.243.254:44133: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
I20260925 14:59:19.565043 14167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:19.569605 14167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
Sep 25 14:59:19 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:19.569890 14167 kserver.cc:163] Server-wide thread pool size limit: 3276
Sep 25 14:59:19 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:19.571590 14167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:19.573566 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:19.559851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:42531)
0925 14:59:19.560103 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:19.560130 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:19.560295 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:19.560704 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:19.560711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:19.561349 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:19.561652 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:19.561668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.563018 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.563027 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:19.565342 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:19.565363 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.566293 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.566303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:19.566574 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.567350 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:19.567385 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:19.573149 (+ 5764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 14:59:19.573771 14032 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:19.560123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:42531 (local address 127.4.243.254:44133)
0925 14:59:19.560258 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:19.560263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:19.560282 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:59:19.560449 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:19.560455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:19.560522 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:19.560628 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:19.561848 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.562840 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.565519 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.566042 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.566194 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.573413 (+ 7219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:42531: BlockingRecv error: recv got EOF from 127.4.243.193:42531 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:59:19.579322 14167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:19.579432 14167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:19.579564 14167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:19.579649 14167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:59:19 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:19 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:59:19.592239 14032 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:19.574934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:60941 (local address 127.4.243.254:44133)
0925 14:59:19.575158 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:19.575166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:19.578894 (+ 3728us) server_negotiation.cc:407] Connection header received
0925 14:59:19.579125 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:19.579134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:19.579218 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:19.579285 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:19.580670 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.582130 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.584937 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.585840 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.585952 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.591885 (+ 5933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:60941: BlockingRecv error: recv got EOF from 127.4.243.193:60941 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:59:19.593000 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:19.574471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:60941)
0925 14:59:19.578808 (+ 4337us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:19.578829 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:19.578983 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:19.579598 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:19.579604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:19.580197 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:19.580457 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:19.580472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.582373 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.582381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:19.584733 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:19.584753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.586198 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.586203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:19.586371 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.587065 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:19.587088 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:19.592560 (+ 5472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4242,"thread_start_us":14972,"threads_started":1}
W20260925 14:59:19.593542 14030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:59:19.632210 14167 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42435
I20260925 14:59:19.632247 14296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42435 every 8 connection(s)
I20260925 14:59:19.633589 14167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:19.635258 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14167
Sep 25 14:59:19 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:19.653652 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:19.636264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.4.243.254:44133)
0925 14:59:19.636517 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:19.636526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:19.637577 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 14:59:19.637667 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:19.637677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:19.637759 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:19.637852 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:19.639800 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.640871 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.642707 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.643410 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.643512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.646567 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:19.646576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:19.646593 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:19.646655 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:19.649387 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:19.650101 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:19.650116 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:19.650129 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:19.650490 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:19.650956 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:19.650964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:19.650970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:19.651786 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:19.652210 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:19.652615 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 14:59:19.653100 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 14:59:19 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:59:20.077440 14166 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.277s user 0.005s sys 0.014s
I20260925 14:59:20.079115 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:18.784246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:57879)
0925 14:59:18.785709 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:18.785753 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:18.789057 (+ 3304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:18.790215 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:18.790230 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:18.791140 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:18.792392 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:18.792420 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.794384 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.794391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:18.796609 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:18.796626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.798517 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.798528 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:18.799016 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:18.800087 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:18.800140 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:20.078191 (+1278051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":472,"threads_started":1}
W20260925 14:59:20.079144 14165 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:18.783311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:57879 (local address 127.4.243.254:44133)
0925 14:59:18.786580 (+ 3269us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:18.786586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:18.786630 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:59:18.789410 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:18.789419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:18.789502 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:18.789608 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:18.792622 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.794035 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.796795 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:18.797863 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:18.798054 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.078633 (+1280579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:57879: BlockingRecv error: recv got EOF from 127.4.243.194:57879 (error 108)
Metrics: {"server-negotiator.queue_time_us":3140,"thread_start_us":577,"threads_started":1}
W20260925 14:59:20.080178 14163 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:44133 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.243.254:44133: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:20.096014 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:20.080965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:36767)
0925 14:59:20.081179 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:20.081199 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:20.081310 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:20.081648 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:20.081652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:20.082008 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:20.082161 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.082169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.083211 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.083217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.084125 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.084135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.084832 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.084838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.085033 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.091873 (+ 6840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:20.091891 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:20.095734 (+ 3843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 14:59:20.096453 14165 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:20.081096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:36767 (local address 127.4.243.254:44133)
0925 14:59:20.081253 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:20.081259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:20.081277 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:59:20.081442 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:20.081446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:20.081521 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:20.081624 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:20.082306 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.083072 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.084271 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.084659 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.084800 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.095908 (+ 11108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:36767: BlockingRecv error: recv got EOF from 127.4.243.194:36767 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:20.107131 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:20.096631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:55505)
0925 14:59:20.097006 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:20.097024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:20.097110 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:20.097581 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:20.097584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:20.097855 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:20.097989 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.097998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.100099 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.100104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.101243 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.101256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.102566 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.102570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.102780 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.103341 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:20.103357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:20.106911 (+ 3554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275}
W20260925 14:59:20.107482 14163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:59:20.107699 14165 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:20.096969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:55505 (local address 127.4.243.254:44133)
0925 14:59:20.097199 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:20.097208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:20.097234 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:59:20.097303 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:20.097311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:20.097408 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:20.097554 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:20.098182 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.099897 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.101399 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.102395 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.102571 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.107274 (+ 4703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:55505: BlockingRecv error: recv got EOF from 127.4.243.194:55505 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:20.606856 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:20.594248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:49441)
0925 14:59:20.594979 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:20.594997 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:20.595181 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:20.595645 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:20.595654 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:20.596439 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:20.596720 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.596737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.597731 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.597739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.600053 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.600075 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.600869 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.600878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.601134 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.601890 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:20.601924 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:20.606475 (+ 4551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":523,"threads_started":1}
W20260925 14:59:20.606910 14165 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:20.594450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:49441 (local address 127.4.243.254:44133)
0925 14:59:20.594863 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:20.594873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:20.595076 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:59:20.595318 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:20.595323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:20.595404 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:20.595516 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:20.596890 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.597581 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.600225 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.600630 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.600744 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.606691 (+ 5947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:49441: BlockingRecv error: recv got EOF from 127.4.243.193:49441 (error 108)
Metrics: {"server-negotiator.queue_time_us":245}
W20260925 14:59:20.837891 14301 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.186s user 0.004s sys 0.009s
W20260925 14:59:20.839184 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:19.637422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:35303 (local address 127.4.243.254:44133)
0925 14:59:19.637862 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:19.637867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:19.639667 (+ 1800us) server_negotiation.cc:407] Connection header received
0925 14:59:19.642279 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:19.642288 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:19.642388 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:19.642564 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:19.644634 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.646545 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.649877 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.650344 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.650477 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.838888 (+1188411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:35303: BlockingRecv error: recv got EOF from 127.4.243.195:35303 (error 108)
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":508,"threads_started":1}
I20260925 14:59:20.839818 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:19.638431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:35303)
0925 14:59:19.639517 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:19.639569 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:19.641965 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:19.642862 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:19.642873 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:19.643575 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:19.644361 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:19.644394 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.646823 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.646831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:19.649684 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:19.649706 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.650586 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.650928 (+ 342us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:19.651146 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.651731 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:19.651755 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:20.838729 (+1186974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":458,"threads_started":1}
W20260925 14:59:20.840673 14297 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:44133 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.243.254:44133: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:20.850082 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:20.841334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:54527)
0925 14:59:20.841544 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:20.841589 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:20.841700 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:20.841996 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:20.842000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:20.842357 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:20.842516 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.842524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.843568 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.843573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.844514 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.844524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.845297 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.845303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.845466 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.846154 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:20.846172 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:20.849880 (+ 3708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 14:59:20.850279 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:20.841468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:54527 (local address 127.4.243.254:44133)
0925 14:59:20.841708 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:20.841714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:20.841733 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:59:20.841802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:20.841807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:20.841869 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:20.841972 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:20.842679 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.843424 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.844655 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.845128 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.845270 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.850069 (+ 4799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:54527: BlockingRecv error: recv got EOF from 127.4.243.195:54527 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:20 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:20.857750 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:20.850747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:59559)
0925 14:59:20.850896 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:20.850916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:20.851002 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:20.851242 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:20.851245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:20.851488 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:20.851628 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.851635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.852554 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.852557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.853403 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:20.853412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.853987 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.853991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:20.854098 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.854670 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:20.854692 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:20.857579 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 14:59:20.858012 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:20.850840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:59559 (local address 127.4.243.254:44133)
0925 14:59:20.851021 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:20.851025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:20.851040 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:59:20.851076 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:20.851079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:20.851127 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:20.851201 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:20.851739 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.852436 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.853510 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:20.853869 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:20.853951 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:20.857733 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:59559: BlockingRecv error: recv got EOF from 127.4.243.195:59559 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 14:59:20.858067 14297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 14:59:21 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:21 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:21.117797 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:21.108073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:38239)
0925 14:59:21.108653 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:21.108669 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:21.108783 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:21.109248 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:21.109251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:21.109681 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:21.109825 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:21.109834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.111256 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.111261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:21.112276 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:21.112288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.113056 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.113062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:21.113219 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.113832 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:21.113850 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:21.117458 (+ 3608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":353,"threads_started":1}
W20260925 14:59:21.117892 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:21.108175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:38239 (local address 127.4.243.254:44133)
0925 14:59:21.108356 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:21.108361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:21.108746 (+ 385us) server_negotiation.cc:407] Connection header received
0925 14:59:21.108982 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:21.108987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:21.109077 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:21.109167 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:21.109998 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.111097 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.112418 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.112861 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.113002 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.117664 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:38239: BlockingRecv error: recv got EOF from 127.4.243.194:38239 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:59:21 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:21 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:21.616803 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:21.607773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:57555)
0925 14:59:21.608257 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:21.608273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:21.608379 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:21.608682 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:21.608685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:21.609061 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:21.609238 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:21.609246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.610262 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.610266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:21.611421 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:21.611433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.612108 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.612112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:21.612340 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.612901 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:21.612924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:21.616445 (+ 3521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":261,"threads_started":1}
W20260925 14:59:21.616858 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:21.607822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:57555 (local address 127.4.243.254:44133)
0925 14:59:21.607988 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:21.607992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:21.608344 (+ 352us) server_negotiation.cc:407] Connection header received
0925 14:59:21.608455 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:21.608458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:21.608515 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:21.608595 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:21.609388 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.610127 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.611545 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.611968 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.612079 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.616640 (+ 4561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:57555: BlockingRecv error: recv got EOF from 127.4.243.193:57555 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 14:59:21 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:21 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:21.867413 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:21.858607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:32793)
0925 14:59:21.859151 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:21.859166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:21.859273 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:21.859567 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:21.859570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:21.860035 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:21.860180 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:21.860188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.861103 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.861106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:21.862001 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:21.862011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.862686 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.862693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:21.862829 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.863563 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:21.863584 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:21.867186 (+ 3602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":291,"threads_started":1}
W20260925 14:59:21.867554 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:21.858716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:32793 (local address 127.4.243.254:44133)
0925 14:59:21.858957 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:21.858965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:21.859248 (+ 283us) server_negotiation.cc:407] Connection header received
0925 14:59:21.859369 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:21.859372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:21.859443 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:21.859520 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:21.860299 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.860999 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.862116 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.862486 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.862570 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.867382 (+ 4812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:32793: BlockingRecv error: recv got EOF from 127.4.243.195:32793 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 14:59:22 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:22 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:22.132165 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:22.118897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:50475)
0925 14:59:22.119363 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:22.119379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:22.119484 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:22.119798 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:22.119801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:22.120163 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:22.120334 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:22.120342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.121441 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.121445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:22.122361 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:22.122372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.123059 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.123064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:22.123282 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:22.124228 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:22.124250 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:22.131810 (+ 7560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":284,"threads_started":1}
W20260925 14:59:22.132315 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:22.118943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:50475 (local address 127.4.243.254:44133)
0925 14:59:22.119151 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:22.119160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:22.119457 (+ 297us) server_negotiation.cc:407] Connection header received
0925 14:59:22.119580 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:22.119584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:22.119674 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:22.119753 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:22.120447 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.121298 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.122477 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.122860 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.122980 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:22.132065 (+ 9085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:50475: BlockingRecv error: recv got EOF from 127.4.243.194:50475 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 14:59:22 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:22 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:22.626740 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:22.617798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:59407)
0925 14:59:22.618270 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:22.618286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:22.618383 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:22.618789 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:22.618792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:22.619125 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:22.619283 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:22.619291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.620459 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.620465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:22.621380 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:22.621391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.622255 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.622261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:22.622436 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:22.623115 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:22.623132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:22.626357 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":243,"threads_started":1}
W20260925 14:59:22.626802 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:22.618058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:59407 (local address 127.4.243.254:44133)
0925 14:59:22.618317 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:22.618322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:22.618339 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:59:22.618548 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:22.618552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:22.618652 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:22.618737 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:22.619433 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.620307 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.621539 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.622109 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.622245 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:22.626543 (+ 4298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:59407: BlockingRecv error: recv got EOF from 127.4.243.193:59407 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 25 14:59:22 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:22 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:22.877635 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:22.868600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:55201)
0925 14:59:22.869143 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:22.869162 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:22.869315 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:22.869720 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:22.869726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:22.870140 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:22.870317 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:22.870326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.871451 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.871459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:22.872377 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:22.872389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.873093 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.873098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:22.873298 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:22.873921 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:22.873938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:22.877277 (+ 3339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":314,"threads_started":1}
W20260925 14:59:22.877688 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:22.868710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:55201 (local address 127.4.243.254:44133)
0925 14:59:22.868920 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:22.868928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:22.869254 (+ 326us) server_negotiation.cc:407] Connection header received
0925 14:59:22.869440 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:22.869445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:22.869561 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:22.869666 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:22.870467 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.871305 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.872544 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:22.872937 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:22.873081 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:22.877454 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:55201: BlockingRecv error: recv got EOF from 127.4.243.195:55201 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 14:59:23 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:23 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:23.141916 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:23.133332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:42257)
0925 14:59:23.133815 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:23.133831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:23.133940 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:23.134261 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:23.134264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:23.134577 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:23.134798 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:23.134806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.135726 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.135729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:23.136594 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:23.136604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.137233 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.137237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:23.137348 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:23.137947 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:23.137963 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:23.141602 (+ 3639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
W20260925 14:59:23.142162 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:23.133434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:42257 (local address 127.4.243.254:44133)
0925 14:59:23.133605 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:23.133610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:23.133903 (+ 293us) server_negotiation.cc:407] Connection header received
0925 14:59:23.134028 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:23.134031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:23.134104 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:23.134176 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:23.134928 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.135587 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.136714 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.137104 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.137200 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:23.141864 (+ 4664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:42257: BlockingRecv error: recv got EOF from 127.4.243.194:42257 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 14:59:23 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:23 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:23.636569 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:23.627955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:41317)
0925 14:59:23.628422 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:23.628438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:23.628541 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:23.628797 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:23.628800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:23.629174 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:23.629325 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:23.629333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.630246 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.630250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:23.631293 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:23.631303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.632211 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.632219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:23.632346 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:23.633002 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:23.633019 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:23.636337 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":270,"threads_started":1}
W20260925 14:59:23.636780 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:23.628044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:41317 (local address 127.4.243.254:44133)
0925 14:59:23.628287 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:23.628294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:23.628517 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:59:23.628620 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:23.628623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:23.628680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:23.628756 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:23.629441 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.630138 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.631449 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.632045 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.632176 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:23.636525 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:41317: BlockingRecv error: recv got EOF from 127.4.243.193:41317 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 14:59:23 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:23 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:23.887197 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:23.878648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:41503)
0925 14:59:23.879141 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:23.879157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:23.879259 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:23.879525 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:23.879527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:23.879872 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:23.880034 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:23.880042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.881008 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.881012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:23.881900 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:23.881911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.882680 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.882685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:23.882793 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:23.883481 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:23.883498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:23.886961 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":253,"threads_started":1}
W20260925 14:59:23.887449 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:23.878694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:41503 (local address 127.4.243.254:44133)
0925 14:59:23.878849 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:23.878853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:23.879224 (+ 371us) server_negotiation.cc:407] Connection header received
0925 14:59:23.879335 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:23.879338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:23.879406 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:23.879483 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:23.880179 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.880892 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.882054 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:23.882480 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:23.882670 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:23.887172 (+ 4502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:41503: BlockingRecv error: recv got EOF from 127.4.243.195:41503 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 14:59:24 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:24 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:24.152946 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:24.142912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:55699)
0925 14:59:24.143485 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:24.143501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:24.143619 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:24.143940 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:24.143943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:24.144613 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:24.144778 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:24.144787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.145954 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.145959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:24.147085 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:24.147098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.147777 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.147782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:24.147903 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.148556 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:24.148574 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:24.152586 (+ 4012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":334,"threads_started":1}
W20260925 14:59:24.153023 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:24.142989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:55699 (local address 127.4.243.254:44133)
0925 14:59:24.143168 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:24.143173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:24.143586 (+ 413us) server_negotiation.cc:407] Connection header received
0925 14:59:24.143700 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:24.143703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:24.143765 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:24.143844 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:24.144932 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.145817 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.147225 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.147614 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.147725 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.152772 (+ 5047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:55699: BlockingRecv error: recv got EOF from 127.4.243.194:55699 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 14:59:24 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:24 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:24.646026 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:24.637505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:39005)
0925 14:59:24.637982 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:24.637996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:24.638164 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:24.638508 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:24.638512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:24.638945 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:24.639089 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:24.639097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.640095 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.640103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:24.641029 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:24.641041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.641763 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.641768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:24.641935 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.642466 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:24.642482 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:24.645804 (+ 3322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
W20260925 14:59:24.646176 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:24.637606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:39005 (local address 127.4.243.254:44133)
0925 14:59:24.637814 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:24.637823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:24.638096 (+ 273us) server_negotiation.cc:407] Connection header received
0925 14:59:24.638284 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:24.638289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:24.638371 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:24.638477 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:24.639236 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.639953 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.641184 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.641620 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.641755 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.645947 (+ 4192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:39005: BlockingRecv error: recv got EOF from 127.4.243.193:39005 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 14:59:24 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:24 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:24.897315 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:24.888118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:41591)
0925 14:59:24.888657 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:24.888672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:24.888774 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:24.889165 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:24.889168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:24.889507 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:24.889668 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:24.889676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.890756 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.890760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:24.891737 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:24.891748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.892426 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.892430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:24.892551 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.893126 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:24.893141 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:24.896969 (+ 3828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":293,"threads_started":1}
W20260925 14:59:24.897403 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:24.888146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:41591 (local address 127.4.243.254:44133)
0925 14:59:24.888309 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:24.888313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:24.888866 (+ 553us) server_negotiation.cc:407] Connection header received
0925 14:59:24.888962 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:24.888966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:24.889032 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:24.889118 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:24.889802 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.890569 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.891880 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.892289 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.892386 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.897172 (+ 4786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:41591: BlockingRecv error: recv got EOF from 127.4.243.195:41591 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 14:59:25 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:25 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:25.166448 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:25.153916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:43191)
0925 14:59:25.154338 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:25.154354 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:25.154458 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:25.154770 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:25.154776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:25.155250 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:25.155390 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:25.155401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.156322 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.156325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:25.157236 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:25.157252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.158197 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.158202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:25.158386 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.162920 (+ 4534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:25.162945 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:25.166220 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":236,"threads_started":1}
W20260925 14:59:25.166729 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:25.154101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:43191 (local address 127.4.243.254:44133)
0925 14:59:25.154319 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:25.154327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:25.154426 (+ 99us) server_negotiation.cc:407] Connection header received
0925 14:59:25.154551 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:25.154554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:25.154634 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:25.154707 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:25.155519 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.156215 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.157403 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.157988 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.158174 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.166444 (+ 8270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:43191: BlockingRecv error: recv got EOF from 127.4.243.194:43191 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 14:59:25 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:25 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:25.655550 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:25.646904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:51923)
0925 14:59:25.647343 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:25.647360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:25.647474 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:25.647774 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:25.647777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:25.648161 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:25.648317 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:25.648325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.649355 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.649359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:25.650270 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:25.650282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.651016 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.651020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:25.651151 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.651831 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:25.651848 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:25.655335 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
W20260925 14:59:25.655745 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:25.647006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:51923 (local address 127.4.243.254:44133)
0925 14:59:25.647244 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:25.647249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:25.647451 (+ 202us) server_negotiation.cc:407] Connection header received
0925 14:59:25.647593 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:25.647596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:25.647656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:25.647731 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:25.648470 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.649234 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.650425 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.650855 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.650956 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.655496 (+ 4540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:51923: BlockingRecv error: recv got EOF from 127.4.243.193:51923 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 14:59:25 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:25 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:25.908038 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:25.898494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:58315)
0925 14:59:25.899165 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:25.899185 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:25.899316 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:25.899743 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:25.899750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:25.900318 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:25.900508 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:25.900519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.901687 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.901691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:25.902758 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:25.902769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.903447 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.903451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:25.903566 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.904325 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:25.904342 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:25.907823 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":413,"threads_started":1}
W20260925 14:59:25.908279 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:25.898523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:58315 (local address 127.4.243.254:44133)
0925 14:59:25.898747 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:25.898754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:25.899293 (+ 539us) server_negotiation.cc:407] Connection header received
0925 14:59:25.899421 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:25.899426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:25.899510 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:25.899639 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:25.900707 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.901561 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.902887 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.903291 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.903392 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.908004 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:58315: BlockingRecv error: recv got EOF from 127.4.243.195:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 14:59:26 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:26 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:26.175993 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:26.167319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:38589)
0925 14:59:26.167798 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:26.167815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:26.167925 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:26.168216 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:26.168219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:26.168591 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:26.168763 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:26.168770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.169748 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.169752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:26.170993 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:26.171005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.171764 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.171769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:26.171883 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:26.172409 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:26.172426 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:26.175779 (+ 3353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":279,"threads_started":1}
W20260925 14:59:26.176204 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:26.167483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:38589 (local address 127.4.243.254:44133)
0925 14:59:26.167733 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:26.167741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:26.167898 (+ 157us) server_negotiation.cc:407] Connection header received
0925 14:59:26.168021 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:26.168025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:26.168084 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:26.168170 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:26.168887 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.169599 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.171117 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.171620 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.171727 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:26.175974 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:38589: BlockingRecv error: recv got EOF from 127.4.243.194:38589 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 14:59:26 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:26 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:26.665817 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:26.656558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:45939)
0925 14:59:26.657133 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:26.657152 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:26.657262 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:26.657541 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:26.657545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:26.657916 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:26.658096 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:26.658104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.659441 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.659447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:26.660769 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:26.660782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.661481 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.661486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:26.661614 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:26.662281 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:26.662298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:26.665584 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":257,"threads_started":1}
W20260925 14:59:26.665977 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:26.656621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:45939 (local address 127.4.243.254:44133)
0925 14:59:26.656841 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:26.656850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:26.657241 (+ 391us) server_negotiation.cc:407] Connection header received
0925 14:59:26.657359 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:26.657363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:26.657422 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:26.657499 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:26.658226 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.659240 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.660897 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.661323 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.661423 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:26.665753 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:45939: BlockingRecv error: recv got EOF from 127.4.243.193:45939 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 14:59:26 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:26 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:26.917538 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:26.908931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:50249)
0925 14:59:26.909493 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:26.909508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:26.909631 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:26.909924 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:26.909927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:26.910324 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:26.910484 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:26.910492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.911682 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.911686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:26.912712 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:26.912723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.913403 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.913420 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:26.913529 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:26.914135 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:26.914151 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:26.917338 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":366,"threads_started":1}
W20260925 14:59:26.917719 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:26.909027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:50249 (local address 127.4.243.254:44133)
0925 14:59:26.909190 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:26.909195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:26.909598 (+ 403us) server_negotiation.cc:407] Connection header received
0925 14:59:26.909732 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:26.909735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:26.909805 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:26.909883 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:26.910655 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.911510 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.912851 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:26.913267 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:26.913363 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:26.917489 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:50249: BlockingRecv error: recv got EOF from 127.4.243.195:50249 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 14:59:27 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:27 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:27.185988 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:27.176915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:58999)
0925 14:59:27.177479 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:27.177495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:27.177664 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:27.177985 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:27.177989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:27.178576 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:27.178811 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:27.178820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.179888 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.179895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:27.180842 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:27.180851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.181532 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.181537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:27.181706 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:27.182276 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:27.182293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:27.185723 (+ 3430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":271,"threads_started":1}
W20260925 14:59:27.186152 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:27.176991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:58999 (local address 127.4.243.254:44133)
0925 14:59:27.177200 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:27.177209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:27.177592 (+ 383us) server_negotiation.cc:407] Connection header received
0925 14:59:27.177796 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:27.177799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:27.177856 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:27.177957 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:27.178955 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.179759 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.180991 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.181378 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.181520 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:27.185928 (+ 4408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:58999: BlockingRecv error: recv got EOF from 127.4.243.194:58999 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 14:59:27 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:27 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:27.675572 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:27.666868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:41057)
0925 14:59:27.667396 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:27.667414 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:27.667527 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:27.667818 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:27.667821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:27.668221 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:27.668381 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:27.668389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.669400 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.669405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:27.670388 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:27.670399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.671224 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.671229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:27.671340 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:27.672064 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:27.672081 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:27.675314 (+ 3233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
W20260925 14:59:27.675773 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:27.666919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:41057 (local address 127.4.243.254:44133)
0925 14:59:27.667185 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:27.667191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:27.667496 (+ 305us) server_negotiation.cc:407] Connection header received
0925 14:59:27.667625 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:27.667628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:27.667691 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:27.667772 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:27.668511 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.669248 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.670522 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.671045 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.671160 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:27.675520 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:41057: BlockingRecv error: recv got EOF from 127.4.243.193:41057 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 14:59:27 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:27 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:27.928210 14324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:27.918530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:36629)
0925 14:59:27.919274 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:27.919293 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:27.919456 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:27.919788 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:27.919793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:27.920199 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:27.920374 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:27.920383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.921398 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.921404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:27.922380 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:27.922390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.923197 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.923203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:27.923376 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:27.924016 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:27.924040 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:27.927968 (+ 3928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":341,"threads_started":1}
W20260925 14:59:27.928295 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:27.918685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:36629 (local address 127.4.243.254:44133)
0925 14:59:27.918881 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:27.918889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:27.919384 (+ 495us) server_negotiation.cc:407] Connection header received
0925 14:59:27.919583 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:27.919588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:27.919656 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:27.919757 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:27.920518 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.921261 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.922533 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:27.923038 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:27.923186 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:27.928122 (+ 4936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:36629: BlockingRecv error: recv got EOF from 127.4.243.195:36629 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 14:59:28 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:28 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:28.196661 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:28.186940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:57255)
0925 14:59:28.187476 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:28.187491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:28.187675 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:28.188022 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:28.188027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:28.188443 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:28.188602 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:28.188610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.189823 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.189829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:28.190836 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:28.190847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.191618 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.191624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:28.191784 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.192350 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:28.192366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:28.196300 (+ 3934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":287,"threads_started":1}
W20260925 14:59:28.196738 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:28.187032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:57255 (local address 127.4.243.254:44133)
0925 14:59:28.187245 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:28.187252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:28.187585 (+ 333us) server_negotiation.cc:407] Connection header received
0925 14:59:28.187809 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:28.187814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:28.187887 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:28.187989 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:28.188756 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.189671 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.190972 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.191442 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.191593 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.196490 (+ 4897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:57255: BlockingRecv error: recv got EOF from 127.4.243.194:57255 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 14:59:28 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:28 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:28.686228 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:28.676492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:40183)
0925 14:59:28.677032 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:28.677048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:28.677154 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:28.677440 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:28.677443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:28.677841 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:28.677977 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:28.677986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.678973 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.678977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:28.679910 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:28.679920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.680592 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.680597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:28.680754 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.681541 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:28.681559 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:28.685901 (+ 4342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":317,"threads_started":1}
W20260925 14:59:28.686383 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:28.676599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:40183 (local address 127.4.243.254:44133)
0925 14:59:28.676790 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:28.676794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:28.677123 (+ 329us) server_negotiation.cc:407] Connection header received
0925 14:59:28.677251 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:28.677255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:28.677313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:28.677394 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:28.678110 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.678862 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.680026 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.680433 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.680528 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.686114 (+ 5586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:40183: BlockingRecv error: recv got EOF from 127.4.243.193:40183 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 14:59:28 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:28 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:28.938175 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:28.929236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:48813)
0925 14:59:28.929733 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:28.929749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:28.929898 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:28.930236 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:28.930244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:28.930736 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:28.930966 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:28.930976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.931970 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.931976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:28.932880 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:28.932890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.933605 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.933612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:28.933780 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.934318 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:28.934334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:28.937837 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":250,"threads_started":1}
W20260925 14:59:28.938263 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:28.929296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:48813 (local address 127.4.243.254:44133)
0925 14:59:28.929510 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:28.929518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:28.929832 (+ 314us) server_negotiation.cc:407] Connection header received
0925 14:59:28.930025 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:28.930031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:28.930097 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:28.930200 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:28.931098 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.931833 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.933017 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.933424 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.933544 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.938007 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:48813: BlockingRecv error: recv got EOF from 127.4.243.195:48813 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 14:59:29 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:29 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:29.207110 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:29.197730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:49459)
0925 14:59:29.198281 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:29.198300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:29.198435 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:29.198842 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:29.198846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:29.199358 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:29.199537 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:29.199545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.200721 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.200725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:29.201692 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:29.201703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.202524 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.202529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:29.202731 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.203301 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:29.203318 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:29.206888 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":279,"threads_started":1}
W20260925 14:59:29.207288 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:29.197863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:49459 (local address 127.4.243.254:44133)
0925 14:59:29.198092 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:29.198099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:29.198397 (+ 298us) server_negotiation.cc:407] Connection header received
0925 14:59:29.198572 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:29.198577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:29.198685 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:29.198793 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:29.199769 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.200571 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.201893 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.202376 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.202477 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.207060 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:49459: BlockingRecv error: recv got EOF from 127.4.243.194:49459 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 14:59:29 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:29 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:29.695586 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:29.687450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:40291)
0925 14:59:29.687954 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:29.687971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:29.688081 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:29.688371 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:29.688374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:29.688720 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:29.688870 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:29.688877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.689805 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.689808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:29.690828 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:29.690839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.691511 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.691516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:29.691643 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.692238 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:29.692254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:29.695358 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
W20260925 14:59:29.695792 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:29.687515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:40291 (local address 127.4.243.254:44133)
0925 14:59:29.687748 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:29.687757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:29.688056 (+ 299us) server_negotiation.cc:407] Connection header received
0925 14:59:29.688176 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:29.688180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:29.688252 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:29.688338 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:29.688989 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.689692 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.690945 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.691348 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.691439 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.695507 (+ 4068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:40291: BlockingRecv error: recv got EOF from 127.4.243.193:40291 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 14:59:29 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:29 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:29.947702 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:29.939208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:49625)
0925 14:59:29.939658 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:29.939674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:29.939777 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:29.940056 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:29.940059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:29.940434 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:29.940596 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:29.940604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.941540 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.941543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:29.942446 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:29.942456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.943118 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.943122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:29.943236 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.943773 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:29.943790 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:29.947371 (+ 3581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
W20260925 14:59:29.947866 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:29.939295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:49625 (local address 127.4.243.254:44133)
0925 14:59:29.939547 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:29.939553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:29.939767 (+ 214us) server_negotiation.cc:407] Connection header received
0925 14:59:29.939873 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:29.939876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:29.939936 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:29.940016 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:29.940724 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.941417 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.942561 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.942999 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.943087 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.947604 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:49625: BlockingRecv error: recv got EOF from 127.4.243.195:49625 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 14:59:30 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:30 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:30.220386 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:30.208048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:49759)
0925 14:59:30.208593 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:30.208609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:30.208717 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:30.209119 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:30.209123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:30.209529 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:30.209744 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:30.209761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.211099 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.211103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:30.212173 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:30.212184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.212839 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.212843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:30.212948 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:30.217020 (+ 4072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:30.217037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:30.220156 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":318,"threads_started":1}
W20260925 14:59:30.220566 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:30.208134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:49759 (local address 127.4.243.254:44133)
0925 14:59:30.208316 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:30.208321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:30.208698 (+ 377us) server_negotiation.cc:407] Connection header received
0925 14:59:30.208837 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:30.208844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:30.208927 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:30.209038 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:30.209893 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.210971 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.212286 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.212714 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.212805 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:30.220327 (+ 7522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:49759: BlockingRecv error: recv got EOF from 127.4.243.194:49759 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 14:59:30 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:30 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:30.705845 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:30.696470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:58807)
0925 14:59:30.696967 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:30.696991 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:30.697149 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:30.697494 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:30.697499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:30.698064 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:30.698293 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:30.698302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.699550 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.699557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:30.700439 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:30.700450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.701191 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.701212 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:30.701399 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:30.701964 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:30.701981 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:30.705628 (+ 3647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
W20260925 14:59:30.706039 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:30.696673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:58807 (local address 127.4.243.254:44133)
0925 14:59:30.696930 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:30.696937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:30.697073 (+ 136us) server_negotiation.cc:407] Connection header received
0925 14:59:30.697291 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:30.697296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:30.697364 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:30.697466 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:30.698439 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.699434 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.700632 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.701032 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.701207 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:30.705817 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:58807: BlockingRecv error: recv got EOF from 127.4.243.193:58807 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 25 14:59:30 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:30 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:30.958143 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:30.948902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:49871)
0925 14:59:30.949484 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:30.949499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:30.949611 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:30.950086 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:30.950091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:30.950651 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:30.950879 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:30.950899 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.952076 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.952080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:30.953079 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:30.953090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.953786 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.953791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:30.953904 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:30.954516 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:30.954533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:30.957917 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":354,"threads_started":1}
W20260925 14:59:30.958328 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:30.948969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:49871 (local address 127.4.243.254:44133)
0925 14:59:30.949156 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:30.949160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:30.949597 (+ 437us) server_negotiation.cc:407] Connection header received
0925 14:59:30.949788 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:30.949792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:30.949901 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:30.950018 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:30.951045 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.951943 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.953220 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:30.953638 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:30.953743 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:30.958087 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:49871: BlockingRecv error: recv got EOF from 127.4.243.195:49871 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 14:59:31 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:31 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:31.230729 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:31.221296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:59625)
0925 14:59:31.221788 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:31.221804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:31.221919 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:31.222244 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:31.222247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:31.222657 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:31.222834 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:31.222842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.223775 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.223779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:31.224614 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:31.224624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.225292 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.225297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:31.225423 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:31.226121 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:31.226140 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:31.230367 (+ 4227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":279,"threads_started":1}
W20260925 14:59:31.230806 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:31.221401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:59625 (local address 127.4.243.254:44133)
0925 14:59:31.221675 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:31.221681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:31.221885 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:59:31.222011 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:31.222015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:31.222071 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:31.222147 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:31.222958 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.223666 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.224734 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.225092 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.225194 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:31.230539 (+ 5345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:59625: BlockingRecv error: recv got EOF from 127.4.243.194:59625 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 25 14:59:31 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:31 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:31.716198 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:31.706782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:45073)
0925 14:59:31.707294 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:31.707309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:31.707457 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:31.707780 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:31.707784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:31.708203 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:31.708365 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:31.708373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.709378 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.709384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:31.710329 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:31.710340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.711232 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.711238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:31.711447 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:31.712086 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:31.712104 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:31.715838 (+ 3734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
W20260925 14:59:31.716280 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:31.706854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:45073 (local address 127.4.243.254:44133)
0925 14:59:31.707069 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:31.707078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:31.707395 (+ 317us) server_negotiation.cc:407] Connection header received
0925 14:59:31.707576 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:31.707581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:31.707645 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:31.707757 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:31.708507 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.709239 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.710479 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.711046 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.711204 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:31.716033 (+ 4829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:45073: BlockingRecv error: recv got EOF from 127.4.243.193:45073 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:59:31 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:31 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:31.968191 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:31.959171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:58681)
0925 14:59:31.959674 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:31.959692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:31.959801 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:31.960072 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:31.960075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:31.960504 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:31.960680 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:31.960689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.961657 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.961661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:31.962560 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:31.962572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.963326 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.963330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:31.963444 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:31.964219 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:31.964241 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:31.967983 (+ 3742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":276,"threads_started":1}
W20260925 14:59:31.968412 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:31.959208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:58681 (local address 127.4.243.254:44133)
0925 14:59:31.959421 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:31.959430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:31.959773 (+ 343us) server_negotiation.cc:407] Connection header received
0925 14:59:31.959894 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:31.959898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:31.959959 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:31.960038 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:31.960798 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.961486 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.962757 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:31.963185 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:31.963284 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:31.968162 (+ 4878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:58681: BlockingRecv error: recv got EOF from 127.4.243.195:58681 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 14:59:32 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:32 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:32.240257 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.231664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:36419)
0925 14:59:32.232175 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.232191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.232314 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.232602 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.232605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.232982 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.233140 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.233148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.234118 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.234122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.235148 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.235159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.235867 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.235871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.236030 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.236644 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.236663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.239988 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":290,"threads_started":1}
W20260925 14:59:32.240442 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.231839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:36419 (local address 127.4.243.254:44133)
0925 14:59:32.232110 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.232116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.232284 (+ 168us) server_negotiation.cc:407] Connection header received
0925 14:59:32.232405 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.232408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.232468 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.232548 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.233268 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.233998 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.235271 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.235696 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.235786 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.240180 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:36419: BlockingRecv error: recv got EOF from 127.4.243.194:36419 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 25 14:59:32 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:32 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:32.725545 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.717105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:49537)
0925 14:59:32.717664 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.717679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.717800 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.718079 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.718082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.718482 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.718694 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.718704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.719611 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.719615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.720510 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.720520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.721151 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.721155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.721287 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.721836 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.721852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.725316 (+ 3464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":353,"threads_started":1}
W20260925 14:59:32.725679 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.717236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:49537 (local address 127.4.243.254:44133)
0925 14:59:32.717440 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.717445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.717752 (+ 307us) server_negotiation.cc:407] Connection header received
0925 14:59:32.717894 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.717898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.717956 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.718039 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.718824 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.719495 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.720629 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.721032 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.721119 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.725476 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:49537: BlockingRecv error: recv got EOF from 127.4.243.193:49537 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 14:59:32 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:32 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:32.978003 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.969265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:39743)
0925 14:59:32.969854 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.969870 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.969981 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.970273 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.970276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.970733 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.970895 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.970903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.971902 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.971906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.972989 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.973001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.973777 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.973782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.973900 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.974569 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.974584 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.977778 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":307,"threads_started":1}
W20260925 14:59:32.978271 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.969432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:39743 (local address 127.4.243.254:44133)
0925 14:59:32.969605 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.969609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.969942 (+ 333us) server_negotiation.cc:407] Connection header received
0925 14:59:32.970074 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.970078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.970135 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.970216 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.971017 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.971739 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.973115 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.973611 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.973697 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.977966 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:39743: BlockingRecv error: recv got EOF from 127.4.243.195:39743 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 14:59:33 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:33 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:33.250128 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.241220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:57117)
0925 14:59:33.241740 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:33.241756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:33.241929 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:33.242358 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:33.242363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:33.242845 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:33.242991 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:33.243000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.244011 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.244016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:33.244941 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:33.244951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.245653 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.245659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:33.245838 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.246398 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:33.246415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:33.249907 (+ 3492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":301,"threads_started":1}
W20260925 14:59:33.250348 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:33.241434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:57117 (local address 127.4.243.254:44133)
0925 14:59:33.241638 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.241645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.241840 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:59:33.242084 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.242089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.242186 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.242328 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:33.243128 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.243862 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.245083 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.245512 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.245652 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.250114 (+ 4462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:57117: BlockingRecv error: recv got EOF from 127.4.243.194:57117 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 14:59:33 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:33 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.193@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:33.735291 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.726737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.193:53069)
0925 14:59:33.727159 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:33.727175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:33.727317 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:33.727647 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:33.727651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:33.728029 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:33.728237 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:33.728246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.729304 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.729316 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:33.730289 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:33.730322 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.731094 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.731098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:33.731230 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.731810 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:33.731826 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:33.735032 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":207,"threads_started":1}
W20260925 14:59:33.735424 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:33.726774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:53069 (local address 127.4.243.254:44133)
0925 14:59:33.726964 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.726972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.727255 (+ 283us) server_negotiation.cc:407] Connection header received
0925 14:59:33.727448 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.727452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.727523 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.727617 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:33.728387 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.729162 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.730434 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.730969 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.731069 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.735200 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.193:53069: BlockingRecv error: recv got EOF from 127.4.243.193:53069 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:59:33 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:33 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.195@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:33.987564 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.978952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.195:38403)
0925 14:59:33.979399 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:33.979415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:33.979574 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:33.979927 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:33.979931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:33.980366 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:33.980510 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:33.980519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.981545 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.981551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:33.982455 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:33.982465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.983148 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.983154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:33.983322 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.983906 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:33.983922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:33.987230 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":248,"threads_started":1}
W20260925 14:59:33.987627 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:33.979026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:38403 (local address 127.4.243.254:44133)
0925 14:59:33.979294 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.979301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.979505 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:59:33.979699 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.979704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.979790 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.979897 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:33.980658 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.981406 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.982619 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.983009 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.983137 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.987417 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.195:38403: BlockingRecv error: recv got EOF from 127.4.243.195:38403 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 25 14:59:34 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:59:34 dist-test-slave-qxh0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.243.194@KRBTEST.COM for oryx/127.4.243.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:59:34.260556 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.251590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44133 (local address 127.4.243.194:45531)
0925 14:59:34.252102 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:34.252129 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:34.252255 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:34.252567 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:34.252570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:34.252908 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:34.253082 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:34.253091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.254505 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.254509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:34.255493 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:34.255504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.256200 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.256204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:34.256352 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:34.256981 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:34.257013 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:34.260318 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.243.254:44133: Server oryx/127.4.243.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":283,"threads_started":1}
W20260925 14:59:34.260761 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:34.251680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:45531 (local address 127.4.243.254:44133)
0925 14:59:34.251875 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:34.251879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:34.252203 (+ 324us) server_negotiation.cc:407] Connection header received
0925 14:59:34.252352 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:34.252356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:34.252430 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:34.252551 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:34.253263 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.254338 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.255616 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.256066 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.256164 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:34.260499 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.243.194:45531: BlockingRecv error: recv got EOF from 127.4.243.194:45531 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:59:34.644569 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 13883
I20260925 14:59:34.657004 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.645636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.4.243.193:36289)
0925 14:59:34.646314 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:34.646321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:34.646348 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:59:34.646426 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:34.646433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:34.646786 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:34.646968 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:34.648344 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.649315 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.650560 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.651167 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.651310 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:34.653800 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:34.653809 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:34.653823 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:34.653855 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:34.656310 (+ 2455us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.243.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 14:59:34.656664 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46800: Request ticket server kudu/127.4.243.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":351,"threads_started":1}
W20260925 14:59:34.657023 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 13883: Not authorized: Client connection negotiation failed: client connection to 127.4.243.193:36289: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.243.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 14:59:34.657119 14345 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46800: Request ticket server kudu/127.4.243.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 14:59:34.657207 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 13883
I20260925 14:59:34.670691 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14034
I20260925 14:59:34.683529 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.671835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.4.243.194:41851)
0925 14:59:34.672449 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:34.672457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:34.672484 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:59:34.672544 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:34.672550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:34.672775 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:34.673002 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:34.674355 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.675552 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.676803 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.677362 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.677541 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:34.680103 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:34.680112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:34.680127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:34.680162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:34.682824 (+ 2662us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.243.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 14:59:34.683187 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42710: Request ticket server kudu/127.4.243.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
W20260925 14:59:34.683531 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14034: Not authorized: Client connection negotiation failed: client connection to 127.4.243.194:41851: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.243.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 14:59:34.683643 14346 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42710: Request ticket server kudu/127.4.243.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 14:59:34.683718 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14034
I20260925 14:59:34.700186 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14167
I20260925 14:59:34.712780 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.701307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.4.243.195:42435)
0925 14:59:34.701869 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:34.701876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:34.701901 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:59:34.701963 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:34.701969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:34.702201 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:34.702415 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:34.703942 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.705034 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.706297 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.706973 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.707124 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:34.709508 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:34.709517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:34.709531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:34.709569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:34.712088 (+ 2519us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.243.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 14:59:34.712423 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37026: Request ticket server kudu/127.4.243.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":273,"threads_started":1}
W20260925 14:59:34.712771 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14167: Not authorized: Client connection negotiation failed: client connection to 127.4.243.195:42435: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.243.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 14:59:34.712883 14347 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37026: Request ticket server kudu/127.4.243.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 14:59:34.712916 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14167
I20260925 14:59:34.725971 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 13814
I20260925 14:59:34.727309 13876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:34.842465 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 13814
2026-09-25T14:59:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21450 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T14:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:34Z Disabled control of system clock
I20260925 14:59:34.915786 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44933
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:59:35.226351 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:35.226742 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:35.226893 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:35.233858 14352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:35.234004 14352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:35.234097 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:35.234192 14352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:35.234282 14352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:35.243233 14352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:44933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:35.244552 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:35.246403 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:35.257752 14357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:35.257797 14358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:35.257723 14360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:35.258473 14352 server_base.cc:1034] running on GCE node
I20260925 14:59:35.259177 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:35.260488 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:35.261679 14352 hybrid_clock.cc:648] HybridClock initialized: now 1790348375261637 us; error 39 us; skew 500 ppm
I20260925 14:59:35.261955 14352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:35.269209 14352 webserver.cc:460] Webserver started at http://127.4.243.254:38417/ using document root <none> and password file <none>
I20260925 14:59:35.269838 14352 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:35.269920 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:35.270179 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:35.272961 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "ef9428c3bf45466588b83c719bd38b0e"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-qxh0"
server_key: "aa0df97bee16ddf827b1aa19a7dbcf35"
server_key_iv: "993c3f6583eef82afa0567813c33679c"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.273720 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "ef9428c3bf45466588b83c719bd38b0e"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-qxh0"
server_key: "aa0df97bee16ddf827b1aa19a7dbcf35"
server_key_iv: "993c3f6583eef82afa0567813c33679c"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.278803 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 14:59:35.282162 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:35.283514 14352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:35.283681 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "ef9428c3bf45466588b83c719bd38b0e"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-qxh0"
server_key: "aa0df97bee16ddf827b1aa19a7dbcf35"
server_key_iv: "993c3f6583eef82afa0567813c33679c"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.283895 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:35.306404 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:35.307178 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:35.307476 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:35.325429 14352 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:44933
I20260925 14:59:35.325450 14417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:44933 every 8 connection(s)
I20260925 14:59:35.326750 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:35.330295 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:35.334887 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14352
I20260925 14:59:35.335086 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:59:35.335660 5071 external_mini_cluster.cc:1428] Setting key 8027d351c43cf7d20d9b80338df1e51f
I20260925 14:59:35.340571 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e: Bootstrap starting.
I20260925 14:59:35.343398 14418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:35.344300 14418 log.cc:824] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:35.347517 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e: No bootstrap required, opened a new log
I20260925 14:59:35.348443 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.337562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.4.243.254:44933)
0925 14:59:35.338497 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.338507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.338551 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:59:35.340109 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.340128 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.340547 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.340932 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.341480 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.342782 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.344093 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.344846 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.345057 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.346182 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.346189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.346203 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:35.346781 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.346859 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:35.346878 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.346970 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.347210 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.347522 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":323,"threads_started":1}
I20260925 14:59:35.355202 14418 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9428c3bf45466588b83c719bd38b0e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44933 } }
I20260925 14:59:35.355569 14418 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:35.355675 14418 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9428c3bf45466588b83c719bd38b0e, State: Initialized, Role: FOLLOWER
I20260925 14:59:35.356353 14418 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9428c3bf45466588b83c719bd38b0e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44933 } }
I20260925 14:59:35.356614 14418 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:35.356750 14418 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:35.356945 14418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:35.359225 14418 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9428c3bf45466588b83c719bd38b0e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44933 } }
I20260925 14:59:35.359746 14418 leader_election.cc:302] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9428c3bf45466588b83c719bd38b0e; no voters:
I20260925 14:59:35.360337 14418 leader_election.cc:288] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:35.360497 14423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:35.361186 14423 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [term 1 LEADER]: Becoming Leader. State: Replica: ef9428c3bf45466588b83c719bd38b0e, State: Running, Role: LEADER
I20260925 14:59:35.361757 14423 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9428c3bf45466588b83c719bd38b0e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44933 } }
I20260925 14:59:35.362452 14418 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:35.364634 14425 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9428c3bf45466588b83c719bd38b0e. Latest consensus state: current_term: 1 leader_uuid: "ef9428c3bf45466588b83c719bd38b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9428c3bf45466588b83c719bd38b0e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44933 } } }
I20260925 14:59:35.364663 14424 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9428c3bf45466588b83c719bd38b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9428c3bf45466588b83c719bd38b0e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 44933 } } }
I20260925 14:59:35.365088 14424 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:35.366242 14429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:35.364966 14425 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:35.372107 14429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:35.380141 14429 catalog_manager.cc:1347] Generated new cluster ID: 0feb590b1e1d443dbd7a0fe8fd21e229
I20260925 14:59:35.380256 14429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:35.403196 14429 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:35.404528 14429 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:35.413810 14429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9428c3bf45466588b83c719bd38b0e: Generated new TSK 0
I20260925 14:59:35.414701 14429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:35.426373 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44933
--builtin_ntp_servers=127.4.243.212:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:59:35.748106 14442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:35.748387 14442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:35.748466 14442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:35.755352 14442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:35.755456 14442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:35.755595 14442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:59:35.764667 14442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:35.765961 14442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:35.767627 14442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:35.779938 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:35.780011 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:35.780025 14450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:35.780828 14442 server_base.cc:1034] running on GCE node
I20260925 14:59:35.781415 14442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:35.782262 14442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:35.783478 14442 hybrid_clock.cc:648] HybridClock initialized: now 1790348375783426 us; error 38 us; skew 500 ppm
I20260925 14:59:35.783775 14442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:35.785838 14442 webserver.cc:460] Webserver started at http://127.4.243.193:44459/ using document root <none> and password file <none>
I20260925 14:59:35.786571 14442 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:35.786751 14442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:35.787101 14442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:35.789775 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "8f858320bd7f4e9b9434c6adcfb50e72"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-qxh0"
server_key: "4203d3920d3c324f1c10acd9f5f21dfe"
server_key_iv: "fb49fd90883646498273e5bf4b4f39ea"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.790516 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "8f858320bd7f4e9b9434c6adcfb50e72"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-qxh0"
server_key: "4203d3920d3c324f1c10acd9f5f21dfe"
server_key_iv: "fb49fd90883646498273e5bf4b4f39ea"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.795204 14442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 14:59:35.798538 14456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:35.799963 14442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:59:35.800137 14442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "8f858320bd7f4e9b9434c6adcfb50e72"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-qxh0"
server_key: "4203d3920d3c324f1c10acd9f5f21dfe"
server_key_iv: "fb49fd90883646498273e5bf4b4f39ea"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.800351 14442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:35.818306 14442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:35.819173 14442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:35.819458 14442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:35.820326 14442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:35.821682 14442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:35.821792 14442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:35.821918 14442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:35.821997 14442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:35.854115 14442 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:45843
I20260925 14:59:35.854182 14568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:45843 every 8 connection(s)
I20260925 14:59:35.855468 14442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:35.857450 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14442
I20260925 14:59:35.857651 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:35.858367 5071 external_mini_cluster.cc:1428] Setting key 6829f9b827161865363a86f3dfd837d4
I20260925 14:59:35.861290 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44933
--builtin_ntp_servers=127.4.243.212:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:59:35.870275 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.857902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:60639 (local address 127.4.243.254:44933)
0925 14:59:35.858441 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.858448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.859735 (+ 1287us) server_negotiation.cc:407] Connection header received
0925 14:59:35.862109 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.862113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.862170 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.862300 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.863717 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.864860 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.867346 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.867752 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.867848 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.869400 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.869438 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.869442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:35.869459 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.869494 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:35.869506 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.869601 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.869852 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.869940 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":251,"threads_started":1}
I20260925 14:59:35.871366 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.858630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44933 (local address 127.4.243.193:60639)
0925 14:59:35.859603 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.859642 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.861832 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.862564 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.862578 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.862717 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.863521 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.863546 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.865065 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.865073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.867199 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.867220 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.867968 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.867977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.868302 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.869088 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:35.869091 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.869127 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:35.869200 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.869599 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.869610 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.869620 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.869829 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.870249 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":279,"threads_started":1}
I20260925 14:59:35.872869 14569 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44933
I20260925 14:59:35.873126 14569 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:35.873848 14569 heartbeater.cc:511] Master 127.4.243.254:44933 requested a full tablet report, sending...
I20260925 14:59:35.875994 14383 ts_manager.cc:194] Registered new tserver with Master: 8f858320bd7f4e9b9434c6adcfb50e72 (127.4.243.193:45843)
I20260925 14:59:35.878199 14383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.193:60639
W20260925 14:59:36.199536 14573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:36.199826 14573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:36.199914 14573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:36.207201 14573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:36.207291 14573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:36.207388 14573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:59:36.215982 14573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:36.217095 14573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:36.218809 14573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:36.232349 14578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:36.232438 14581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:36.232832 14573 server_base.cc:1034] running on GCE node
W20260925 14:59:36.232438 14579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:36.233525 14573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:36.234427 14573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:36.235651 14573 hybrid_clock.cc:648] HybridClock initialized: now 1790348376235593 us; error 46 us; skew 500 ppm
I20260925 14:59:36.235987 14573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:36.238309 14573 webserver.cc:460] Webserver started at http://127.4.243.194:41591/ using document root <none> and password file <none>
I20260925 14:59:36.239101 14573 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:36.239193 14573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:36.239547 14573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:36.242457 14573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "bbc4f9d892524826ad7fc34a0886b42e"
format_stamp: "Formatted at 2026-09-25 14:59:36 on dist-test-slave-qxh0"
server_key: "9ea102b5ce77bd517dfc03be70ea483b"
server_key_iv: "24ac3635bd9f5ea05840d90d16ef9ecc"
server_key_version: "encryptionkey@0"
I20260925 14:59:36.243319 14573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "bbc4f9d892524826ad7fc34a0886b42e"
format_stamp: "Formatted at 2026-09-25 14:59:36 on dist-test-slave-qxh0"
server_key: "9ea102b5ce77bd517dfc03be70ea483b"
server_key_iv: "24ac3635bd9f5ea05840d90d16ef9ecc"
server_key_version: "encryptionkey@0"
I20260925 14:59:36.248391 14573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:59:36.251721 14587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:36.253024 14573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 14:59:36.253209 14573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "bbc4f9d892524826ad7fc34a0886b42e"
format_stamp: "Formatted at 2026-09-25 14:59:36 on dist-test-slave-qxh0"
server_key: "9ea102b5ce77bd517dfc03be70ea483b"
server_key_iv: "24ac3635bd9f5ea05840d90d16ef9ecc"
server_key_version: "encryptionkey@0"
I20260925 14:59:36.253444 14573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:36.272037 14573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:36.272780 14573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:36.273077 14573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:36.273979 14573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:36.275430 14573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:36.275565 14573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:36.275699 14573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:36.275781 14573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:36.308113 14573 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:43403
I20260925 14:59:36.308162 14699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:43403 every 8 connection(s)
I20260925 14:59:36.309357 14573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:36.316648 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14573
I20260925 14:59:36.316847 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:36.317255 5071 external_mini_cluster.cc:1428] Setting key b48b289fe45d977b57d629945ac06211
I20260925 14:59:36.319835 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44933
--builtin_ntp_servers=127.4.243.212:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:59:36.321931 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.311800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:56867 (local address 127.4.243.254:44933)
0925 14:59:36.311969 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.311973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.313333 (+ 1360us) server_negotiation.cc:407] Connection header received
0925 14:59:36.315120 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.315125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.315214 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.315333 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:36.316374 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.317195 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.319167 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.319564 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.319697 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.321123 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:36.321128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:36.321131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:36.321148 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:36.321181 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:36.321192 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:36.321284 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.321520 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.321610 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:59:36.323024 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.312324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44933 (local address 127.4.243.194:56867)
0925 14:59:36.313184 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:36.313218 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:36.314859 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:36.315557 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:36.315565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:36.315650 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:36.316198 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:36.316213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.317358 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.317364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:36.319002 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:36.319018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.319793 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.319800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:36.320079 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.320820 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:36.320823 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:36.320860 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:36.320934 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:36.321284 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:36.321297 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:36.321307 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:59:36.321514 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 14:59:36.321920 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":257,"threads_started":1}
I20260925 14:59:36.324344 14700 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44933
I20260925 14:59:36.324658 14700 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:36.325378 14700 heartbeater.cc:511] Master 127.4.243.254:44933 requested a full tablet report, sending...
I20260925 14:59:36.326870 14383 ts_manager.cc:194] Registered new tserver with Master: bbc4f9d892524826ad7fc34a0886b42e (127.4.243.194:43403)
I20260925 14:59:36.327924 14383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.194:56867
W20260925 14:59:36.642005 14703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:36.642349 14703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:36.642511 14703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:36.649624 14703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:36.649755 14703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:36.649940 14703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:59:36.659029 14703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:36.660229 14703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:36.661885 14703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:36.674831 14711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:36.674805 14708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:36.674902 14709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:36.675499 14703 server_base.cc:1034] running on GCE node
I20260925 14:59:36.676086 14703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:36.677016 14703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:36.678241 14703 hybrid_clock.cc:648] HybridClock initialized: now 1790348376678192 us; error 43 us; skew 500 ppm
I20260925 14:59:36.678579 14703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:36.680769 14703 webserver.cc:460] Webserver started at http://127.4.243.195:33469/ using document root <none> and password file <none>
I20260925 14:59:36.681461 14703 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:36.681579 14703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:36.681910 14703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:36.684749 14703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "e48c2e5b0c564bb98b157814354a56b6"
format_stamp: "Formatted at 2026-09-25 14:59:36 on dist-test-slave-qxh0"
server_key: "65402e2ff48347bb31fa69fee446585a"
server_key_iv: "f6a2874cdac7893c089f4f9885a77fcb"
server_key_version: "encryptionkey@0"
I20260925 14:59:36.685523 14703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "e48c2e5b0c564bb98b157814354a56b6"
format_stamp: "Formatted at 2026-09-25 14:59:36 on dist-test-slave-qxh0"
server_key: "65402e2ff48347bb31fa69fee446585a"
server_key_iv: "f6a2874cdac7893c089f4f9885a77fcb"
server_key_version: "encryptionkey@0"
I20260925 14:59:36.690445 14703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 14:59:36.693547 14717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:36.694803 14703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:36.694964 14703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "e48c2e5b0c564bb98b157814354a56b6"
format_stamp: "Formatted at 2026-09-25 14:59:36 on dist-test-slave-qxh0"
server_key: "65402e2ff48347bb31fa69fee446585a"
server_key_iv: "f6a2874cdac7893c089f4f9885a77fcb"
server_key_version: "encryptionkey@0"
I20260925 14:59:36.695151 14703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:36.716169 14703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:36.716948 14703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:36.717247 14703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:36.718165 14703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:36.719676 14703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:36.719799 14703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:36.719939 14703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:36.720029 14703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:36.751886 14703 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:40823
I20260925 14:59:36.751967 14829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:40823 every 8 connection(s)
I20260925 14:59:36.753091 14703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:36.761014 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14703
I20260925 14:59:36.761241 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:36.761849 5071 external_mini_cluster.cc:1428] Setting key 4f6a0405dea96d911bd043d4ce6c7270
I20260925 14:59:36.766237 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.755652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:37439 (local address 127.4.243.254:44933)
0925 14:59:36.755843 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.755847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.757336 (+ 1489us) server_negotiation.cc:407] Connection header received
0925 14:59:36.759213 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.759218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.759291 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.759384 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:36.760516 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.761396 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.763487 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.764102 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.764197 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.765622 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:36.765626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:36.765630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:36.765645 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:36.765674 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:36.765685 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:36.765753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.765948 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.766015 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:59:36.767242 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.756189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:44933 (local address 127.4.243.195:37439)
0925 14:59:36.757191 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:36.757224 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:36.758938 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:36.759653 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:36.759661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:36.759767 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:36.760342 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:36.760358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.761550 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.761558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:36.763333 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:36.763365 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.764309 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.764319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:36.764603 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.765347 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:36.765349 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:36.765388 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:36.765464 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:36.765767 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:36.765774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:36.765779 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:59:36.765920 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 14:59:36.766188 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":358,"threads_started":1}
I20260925 14:59:36.768420 14830 heartbeater.cc:347] Connected to a master server at 127.4.243.254:44933
I20260925 14:59:36.768695 14830 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:36.769486 14830 heartbeater.cc:511] Master 127.4.243.254:44933 requested a full tablet report, sending...
I20260925 14:59:36.770804 14383 ts_manager.cc:194] Registered new tserver with Master: e48c2e5b0c564bb98b157814354a56b6 (127.4.243.195:40823)
I20260925 14:59:36.771600 14383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.195:37439
I20260925 14:59:36.777130 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:59:36.782768 14571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:36.781327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.4.243.254:44933)
0925 14:59:36.781532 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.781542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.781923 (+ 381us) server_negotiation.cc:407] Connection header received
0925 14:59:36.782036 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.782039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.782104 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.782181 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:36.782543 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46628: BlockingRecv error: recv got EOF from 127.0.0.1:46628 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:59:36.783541 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14442
I20260925 14:59:36.790437 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.784324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.4.243.193:45843)
0925 14:59:36.784817 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.784823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.784852 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:59:36.784907 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.784913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.785078 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.785195 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:36.785539 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.786458 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.787764 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.788326 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.788422 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.789264 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:36.789271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:36.789282 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:36.789791 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:36.789830 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:36.789861 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:36.789945 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.790063 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.790168 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260925 14:59:36.791520 14564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:36.871709 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14442
I20260925 14:59:36.885329 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14573
I20260925 14:59:36.892948 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.886355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.4.243.194:43403)
0925 14:59:36.886953 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.886960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.886985 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:59:36.887043 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.887050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.887279 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.887415 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:36.887790 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.888774 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.889909 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.890433 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.890529 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.891532 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:36.891540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:36.891553 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:36.892128 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:36.892171 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:36.892190 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:36.892276 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.892422 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.892598 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":345,"threads_started":1}
I20260925 14:59:36.894160 14695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:36.978494 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14573
I20260925 14:59:36.995000 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14703
I20260925 14:59:37.002952 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.996033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.4.243.195:40823)
0925 14:59:36.996650 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.996657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.996687 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:59:36.996751 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.996757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.996980 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.997102 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:36.997549 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.998654 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.999913 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:37.000476 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:37.000572 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:37.001534 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:37.001542 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:37.001556 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:37.002126 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:37.002171 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:37.002189 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:37.002278 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:37.002439 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:59:37.002576 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":313,"threads_started":1}
I20260925 14:59:37.004220 14825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:37.084738 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14703
I20260925 14:59:37.100275 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14352
I20260925 14:59:37.101424 14413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:37.204177 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14352
2026-09-25T14:59:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2356 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T14:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:37Z Disabled control of system clock
I20260925 14:59:37.272682 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33253
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:59:37.595993 14849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:37.596256 14849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:37.596333 14849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:37.603302 14849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:37.603410 14849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:37.603446 14849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:37.603507 14849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:37.603542 14849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:37.612239 14849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:33253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:37.613448 14849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:37.615144 14849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:37.626585 14855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:37.626667 14857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:37.626667 14854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:37.627419 14849 server_base.cc:1034] running on GCE node
I20260925 14:59:37.628158 14849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:37.629382 14849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:37.630573 14849 hybrid_clock.cc:648] HybridClock initialized: now 1790348377630516 us; error 47 us; skew 500 ppm
I20260925 14:59:37.630897 14849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:37.632946 14849 webserver.cc:460] Webserver started at http://127.4.243.254:36435/ using document root <none> and password file <none>
I20260925 14:59:37.633543 14849 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:37.633620 14849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:37.633865 14849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:37.636483 14849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "ed31135ebc3848c998b433e54f36de0b"
format_stamp: "Formatted at 2026-09-25 14:59:37 on dist-test-slave-qxh0"
server_key: "c2ed538e1398c1b999a558ce0303e3d9"
server_key_iv: "88ba6e927a56c4f582d82ba9c8349312"
server_key_version: "encryptionkey@0"
I20260925 14:59:37.637166 14849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "ed31135ebc3848c998b433e54f36de0b"
format_stamp: "Formatted at 2026-09-25 14:59:37 on dist-test-slave-qxh0"
server_key: "c2ed538e1398c1b999a558ce0303e3d9"
server_key_iv: "88ba6e927a56c4f582d82ba9c8349312"
server_key_version: "encryptionkey@0"
I20260925 14:59:37.642066 14849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 14:59:37.645347 14863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:37.646643 14849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:37.646831 14849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "ed31135ebc3848c998b433e54f36de0b"
format_stamp: "Formatted at 2026-09-25 14:59:37 on dist-test-slave-qxh0"
server_key: "c2ed538e1398c1b999a558ce0303e3d9"
server_key_iv: "88ba6e927a56c4f582d82ba9c8349312"
server_key_version: "encryptionkey@0"
I20260925 14:59:37.647012 14849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:37.652436 14849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:37.653097 14849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:37.653327 14849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:37.670827 14849 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:33253
I20260925 14:59:37.670866 14914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:33253 every 8 connection(s)
I20260925 14:59:37.672058 14849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:37.675591 14915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:37.680866 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14849
I20260925 14:59:37.681070 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:59:37.681700 5071 external_mini_cluster.cc:1428] Setting key e8c779a439b2eb93b38f72e42929c9f3
I20260925 14:59:37.686808 14915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b: Bootstrap starting.
I20260925 14:59:37.690618 14915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:37.691043 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:37.683739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.4.243.254:33253)
0925 14:59:37.684876 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:37.684890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:37.684940 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:59:37.687154 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:37.687193 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:37.687694 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:37.688078 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:37.689089 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:37.689096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:37.689110 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:37.689125 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:37.689195 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:37.689214 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:37.689501 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:37.689622 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 14:59:37.689869 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":340,"threads_started":1}
I20260925 14:59:37.691807 14915 log.cc:824] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:37.694491 14915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b: No bootstrap required, opened a new log
I20260925 14:59:37.701874 14915 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed31135ebc3848c998b433e54f36de0b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33253 } }
I20260925 14:59:37.702201 14915 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:37.702303 14915 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed31135ebc3848c998b433e54f36de0b, State: Initialized, Role: FOLLOWER
I20260925 14:59:37.702901 14915 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed31135ebc3848c998b433e54f36de0b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33253 } }
I20260925 14:59:37.703120 14915 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:37.703245 14915 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:37.703414 14915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:37.705354 14915 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed31135ebc3848c998b433e54f36de0b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33253 } }
I20260925 14:59:37.705861 14915 leader_election.cc:302] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed31135ebc3848c998b433e54f36de0b; no voters:
I20260925 14:59:37.706416 14915 leader_election.cc:288] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:37.706542 14920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:37.707317 14920 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [term 1 LEADER]: Becoming Leader. State: Replica: ed31135ebc3848c998b433e54f36de0b, State: Running, Role: LEADER
I20260925 14:59:37.708220 14920 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed31135ebc3848c998b433e54f36de0b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33253 } }
I20260925 14:59:37.708475 14915 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:37.710922 14921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed31135ebc3848c998b433e54f36de0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed31135ebc3848c998b433e54f36de0b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33253 } } }
I20260925 14:59:37.710964 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed31135ebc3848c998b433e54f36de0b. Latest consensus state: current_term: 1 leader_uuid: "ed31135ebc3848c998b433e54f36de0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed31135ebc3848c998b433e54f36de0b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33253 } } }
I20260925 14:59:37.711299 14921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:37.711385 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:37.712205 14928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:37.718353 14928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:37.727234 14928 catalog_manager.cc:1347] Generated new cluster ID: 3ed07366e88749e2bd7645d19ebf37ea
I20260925 14:59:37.727342 14928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:37.740824 14928 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:37.741250 14928 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:37.762260 14928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed31135ebc3848c998b433e54f36de0b: Generated new TSK 0
I20260925 14:59:37.763348 14928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:37.775171 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33253
--builtin_ntp_servers=127.4.243.212:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:59:38.100978 14939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:38.101262 14939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:38.101450 14939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:38.108397 14939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:38.108511 14939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:38.108690 14939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:59:38.117534 14939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:38.118834 14939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:38.120612 14939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:38.133834 14944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:38.133790 14947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:38.133733 14945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:38.134778 14939 server_base.cc:1034] running on GCE node
I20260925 14:59:38.135426 14939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:38.136318 14939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:38.137513 14939 hybrid_clock.cc:648] HybridClock initialized: now 1790348378137463 us; error 43 us; skew 500 ppm
I20260925 14:59:38.137825 14939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:38.139981 14939 webserver.cc:460] Webserver started at http://127.4.243.193:41609/ using document root <none> and password file <none>
I20260925 14:59:38.140656 14939 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:38.140765 14939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:38.141086 14939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:38.145545 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "30a23cc3c216411198b4b0323d793293"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "89c85325dd652f40103a665385f1b3a1"
server_key_iv: "479a66968902437575e8de6d43f870fa"
server_key_version: "encryptionkey@0"
I20260925 14:59:38.146351 14939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "30a23cc3c216411198b4b0323d793293"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "89c85325dd652f40103a665385f1b3a1"
server_key_iv: "479a66968902437575e8de6d43f870fa"
server_key_version: "encryptionkey@0"
I20260925 14:59:38.151664 14939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 14:59:38.154883 14953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:38.156165 14939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:59:38.156370 14939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "30a23cc3c216411198b4b0323d793293"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "89c85325dd652f40103a665385f1b3a1"
server_key_iv: "479a66968902437575e8de6d43f870fa"
server_key_version: "encryptionkey@0"
I20260925 14:59:38.156582 14939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:38.162292 14939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:38.163117 14939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:38.163400 14939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:38.164280 14939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:38.165643 14939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:38.165760 14939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:38.165891 14939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:38.165979 14939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:38.198338 14939 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:43841
I20260925 14:59:38.198388 15065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:43841 every 8 connection(s)
I20260925 14:59:38.199707 14939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:38.208158 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:38.202237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:59023 (local address 127.4.243.254:33253)
0925 14:59:38.202711 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:38.202716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:38.203882 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 14:59:38.205609 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:38.205615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:38.205685 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:38.205856 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:38.207375 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:38.207380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:38.207385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:38.207403 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:38.207453 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:38.207466 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:38.207621 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:38.207800 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:59:38.207878 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":246,"threads_started":1}
I20260925 14:59:38.208554 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 14939
I20260925 14:59:38.208745 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:38.209146 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:38.202766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33253 (local address 127.4.243.193:59023)
0925 14:59:38.203712 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:38.203749 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:38.205336 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:38.206210 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:38.206219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:38.206315 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:38.207085 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:38.207088 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:38.207122 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:38.207202 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:38.207564 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:38.207574 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:38.207601 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:59:38.207880 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 14:59:38.208149 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":311,"threads_started":1}
I20260925 14:59:38.209487 5071 external_mini_cluster.cc:1428] Setting key a3e2790ff74f056a3a104c79afdb998b
I20260925 14:59:38.210798 15066 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33253
I20260925 14:59:38.211081 15066 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:38.211755 15066 heartbeater.cc:511] Master 127.4.243.254:33253 requested a full tablet report, sending...
I20260925 14:59:38.212199 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33253
--builtin_ntp_servers=127.4.243.212:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 14:59:38.213841 14880 ts_manager.cc:194] Registered new tserver with Master: 30a23cc3c216411198b4b0323d793293 (127.4.243.193:43841)
W20260925 14:59:38.527019 15070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:38.527278 15070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:38.527361 15070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:38.534415 15070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:38.534495 15070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:38.534649 15070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:59:38.543274 15070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:38.544438 15070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:38.546105 15070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:38.558434 15075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:38.558389 15078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:38.558470 15076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:38.559696 15070 server_base.cc:1034] running on GCE node
I20260925 14:59:38.560348 15070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:38.561278 15070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:38.562543 15070 hybrid_clock.cc:648] HybridClock initialized: now 1790348378562481 us; error 52 us; skew 500 ppm
I20260925 14:59:38.562889 15070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:38.565011 15070 webserver.cc:460] Webserver started at http://127.4.243.194:36653/ using document root <none> and password file <none>
I20260925 14:59:38.565649 15070 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:38.565732 15070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:38.566004 15070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:38.568641 15070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "f7535fe976494d97bc6c72bd61638a9f"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "bc5563f05e24372f99d0dd4fba44c022"
server_key_iv: "282f7b1f4b7b371099b508fe45e2dec0"
server_key_version: "encryptionkey@0"
I20260925 14:59:38.569334 15070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "f7535fe976494d97bc6c72bd61638a9f"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "bc5563f05e24372f99d0dd4fba44c022"
server_key_iv: "282f7b1f4b7b371099b508fe45e2dec0"
server_key_version: "encryptionkey@0"
I20260925 14:59:38.574105 15070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 14:59:38.577606 15084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:38.578912 15070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:38.579074 15070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "f7535fe976494d97bc6c72bd61638a9f"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "bc5563f05e24372f99d0dd4fba44c022"
server_key_iv: "282f7b1f4b7b371099b508fe45e2dec0"
server_key_version: "encryptionkey@0"
I20260925 14:59:38.579280 15070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:38.584611 15070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:38.585335 15070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:38.585613 15070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:38.586447 15070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:38.587773 15070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:38.587891 15070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:38.588038 15070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:38.588124 15070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:38.619626 15070 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:45807
I20260925 14:59:38.619643 15196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:45807 every 8 connection(s)
I20260925 14:59:38.620934 15070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:38.622931 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 15070
I20260925 14:59:38.623085 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:38.623569 5071 external_mini_cluster.cc:1428] Setting key 967f49da740e1d05b3faf765906eea08
I20260925 14:59:38.626502 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33253
--builtin_ntp_servers=127.4.243.212:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 14:59:38.630137 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:38.623621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:33285 (local address 127.4.243.254:33253)
0925 14:59:38.623816 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:38.623822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:38.625448 (+ 1626us) server_negotiation.cc:407] Connection header received
0925 14:59:38.627985 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:38.627988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:38.628045 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:38.628126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:38.629526 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:38.629530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:38.629533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:38.629547 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:38.629575 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:38.629586 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:38.629692 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:38.629897 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 14:59:38.629940 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:59:38.631327 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:38.624274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33253 (local address 127.4.243.194:33285)
0925 14:59:38.625237 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:38.625316 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:38.627700 (+ 2384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:38.628389 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:38.628402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:38.628531 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:38.629201 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:38.629206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:38.629253 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:38.629359 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:38.629691 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:38.629702 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:38.629712 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:59:38.629968 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 14:59:38.630192 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":295,"threads_started":1}
I20260925 14:59:38.632627 15197 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33253
I20260925 14:59:38.632891 15197 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:38.633419 15197 heartbeater.cc:511] Master 127.4.243.254:33253 requested a full tablet report, sending...
I20260925 14:59:38.634797 14880 ts_manager.cc:194] Registered new tserver with Master: f7535fe976494d97bc6c72bd61638a9f (127.4.243.194:45807)
W20260925 14:59:38.956918 15200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:38.957195 15200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:38.957333 15200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:38.964915 15200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:38.965034 15200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:38.965210 15200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:59:38.974166 15200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:38.975417 15200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:38.977099 15200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:38.990018 15205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:38.989993 15206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:38.989955 15208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:38.990724 15200 server_base.cc:1034] running on GCE node
I20260925 14:59:38.991272 15200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:38.992115 15200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:38.993323 15200 hybrid_clock.cc:648] HybridClock initialized: now 1790348378993284 us; error 62 us; skew 500 ppm
I20260925 14:59:38.993629 15200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 14:59:38.995769 15200 webserver.cc:460] Webserver started at http://127.4.243.195:41663/ using document root <none> and password file <none>
I20260925 14:59:38.996399 15200 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:38.996520 15200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:38.996837 15200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:38.999670 15200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "4ab3bc4683384f37961328ef814b7588"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "d9a4ce5fa30c86a210ebf4330b362453"
server_key_iv: "5d6e94f69707cc7de053abc36bbc6d8a"
server_key_version: "encryptionkey@0"
I20260925 14:59:39.000483 15200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "4ab3bc4683384f37961328ef814b7588"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "d9a4ce5fa30c86a210ebf4330b362453"
server_key_iv: "5d6e94f69707cc7de053abc36bbc6d8a"
server_key_version: "encryptionkey@0"
I20260925 14:59:39.005512 15200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 14:59:39.008895 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:39.010242 15200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:39.010421 15200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "4ab3bc4683384f37961328ef814b7588"
format_stamp: "Formatted at 2026-09-25 14:59:38 on dist-test-slave-qxh0"
server_key: "d9a4ce5fa30c86a210ebf4330b362453"
server_key_iv: "5d6e94f69707cc7de053abc36bbc6d8a"
server_key_version: "encryptionkey@0"
I20260925 14:59:39.010644 15200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:39.016095 15200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:39.016795 15200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:39.017037 15200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:39.017889 15200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:39.019207 15200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:39.019313 15200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:39.019441 15200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:39.019531 15200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:39.050127 15200 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:36625
I20260925 14:59:39.050169 15326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:36625 every 8 connection(s)
I20260925 14:59:39.051528 15200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:39.059345 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:39.054045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:49873 (local address 127.4.243.254:33253)
0925 14:59:39.054213 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:39.054218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:39.055633 (+ 1415us) server_negotiation.cc:407] Connection header received
0925 14:59:39.057249 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:39.057254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:39.057325 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:39.057430 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:39.058664 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:39.058669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:39.058674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:39.058692 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:39.058726 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:39.058737 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:39.058859 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:39.059083 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:59:39.059127 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:59:39.059698 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 15200
I20260925 14:59:39.059854 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:39.060417 5071 external_mini_cluster.cc:1428] Setting key f38ee4758926ac883ac1de19211c0e79
I20260925 14:59:39.060580 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:39.054567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33253 (local address 127.4.243.195:49873)
0925 14:59:39.055460 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:39.055519 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:39.056997 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:39.057633 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:39.057641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:39.057723 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:39.058310 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:39.058313 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:39.058345 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:39.058456 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:39.058889 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:39.058900 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:39.058915 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:59:39.059127 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 14:59:39.059355 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":252,"threads_started":1}
I20260925 14:59:39.061955 15327 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33253
I20260925 14:59:39.062230 15327 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:39.062893 15327 heartbeater.cc:511] Master 127.4.243.254:33253 requested a full tablet report, sending...
I20260925 14:59:39.064200 14879 ts_manager.cc:194] Registered new tserver with Master: 4ab3bc4683384f37961328ef814b7588 (127.4.243.195:36625)
I20260925 14:59:39.075408 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:59:39.081264 15068 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:39.079803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.4.243.254:33253)
0925 14:59:39.080009 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:39.080017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:39.080390 (+ 373us) server_negotiation.cc:407] Connection header received
0925 14:59:39.080585 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:39.080590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:39.080647 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:39.080750 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:39.081079 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46972: BlockingRecv error: recv got EOF from 127.0.0.1:46972 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:59:39.082232 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14939
I20260925 14:59:39.085593 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:39.083216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.4.243.193:43841)
0925 14:59:39.083723 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:39.083739 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:39.083765 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:59:39.083819 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:39.083826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:39.084032 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:39.084153 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:39.085009 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:39.085016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:39.085028 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:39.085040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:39.085082 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:39.085099 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:39.085198 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:39.085284 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:59:39.085374 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260925 14:59:39.086762 15061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:39.165552 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14939
I20260925 14:59:39.181331 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 15070
I20260925 14:59:39.185252 15338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:39.182440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.4.243.194:45807)
0925 14:59:39.182982 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:39.182989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:39.183015 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:59:39.183104 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:39.183115 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:39.183344 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:39.183513 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:39.184558 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:39.184565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:39.184575 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:39.184591 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:39.184627 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:39.184640 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:39.184779 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:39.184911 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:59:39.185011 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":260,"threads_started":1}
I20260925 14:59:39.186439 15192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:39.265799 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 15070
I20260925 14:59:39.281271 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 15200
I20260925 14:59:39.285531 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:39.282248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.4.243.195:36625)
0925 14:59:39.282943 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:39.282950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:39.282980 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:59:39.283042 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:39.283049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:39.283268 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:39.283454 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:39.284674 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:39.284681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:39.284693 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:39.284706 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:39.284743 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:39.284757 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:39.284946 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:39.285053 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:59:39.285153 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":300,"threads_started":1}
I20260925 14:59:39.286762 15322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:39.369402 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 15200
I20260925 14:59:39.384822 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 14849
I20260925 14:59:39.386026 14910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:39.484231 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 14849
2026-09-25T14:59:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2279 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:39 dist-test-slave-qxh0 krb5kdc[15345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:39 dist-test-slave-qxh0 krb5kdc[15345](info): set up 2 sockets
Sep 25 14:59:39 dist-test-slave-qxh0 krb5kdc[15345](info): commencing operation
krb5kdc: starting...
W20260925 14:59:40.567631 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:59:40 dist-test-slave-qxh0 krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 14:59:40.774976 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:39713
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:39713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:41.089839 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:41.090147 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:41.090301 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:41.097376 15358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:41.097474 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:41.097589 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:41.097687 15358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:41.097779 15358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:41.107161 15358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:39713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:39713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:41.108394 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:41.109623 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:41.121088 15364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:41.121338 15363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:41.121295 15366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:41.122308 15358 server_base.cc:1034] running on GCE node
I20260925 14:59:41.123248 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:41.124706 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:41.125929 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790348381125871 us; error 56 us; skew 500 ppm
I20260925 14:59:41.126271 15358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:41 dist-test-slave-qxh0 krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348381, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:41.134099 15358 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 14:59:41.135563 15358 webserver.cc:460] Webserver started at http://127.4.243.254:38123/ using document root <none> and password file <none>
I20260925 14:59:41.136215 15358 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:41.136353 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:41.136642 15358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:41.139576 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9"
format_stamp: "Formatted at 2026-09-25 14:59:41 on dist-test-slave-qxh0"
server_key: "7c3fca4abfb47d231a63c47fff20a25b"
server_key_iv: "00f4cbbd1f748bf87385c23f264326a0"
server_key_version: "encryptionkey@0"
I20260925 14:59:41.140296 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9"
format_stamp: "Formatted at 2026-09-25 14:59:41 on dist-test-slave-qxh0"
server_key: "7c3fca4abfb47d231a63c47fff20a25b"
server_key_iv: "00f4cbbd1f748bf87385c23f264326a0"
server_key_version: "encryptionkey@0"
I20260925 14:59:41.145081 15358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 14:59:41.148231 15373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:41.149452 15358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:41.149653 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9"
format_stamp: "Formatted at 2026-09-25 14:59:41 on dist-test-slave-qxh0"
server_key: "7c3fca4abfb47d231a63c47fff20a25b"
server_key_iv: "00f4cbbd1f748bf87385c23f264326a0"
server_key_version: "encryptionkey@0"
I20260925 14:59:41.149833 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:41.183106 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:41.187189 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:41.187482 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:41.205675 15358 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:39713
I20260925 14:59:41.205729 15424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:39713 every 8 connection(s)
I20260925 14:59:41.207156 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:41.210762 15425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:41.214514 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 15358
I20260925 14:59:41.214756 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 14:59:41.215386 5071 external_mini_cluster.cc:1428] Setting key 5615e060959e57093049ee55d50a8871
I20260925 14:59:41.222658 15425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9: Bootstrap starting.
I20260925 14:59:41.225672 15425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:41.226706 15425 log.cc:824] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:41.229336 15425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9: No bootstrap required, opened a new log
Sep 25 14:59:41 dist-test-slave-qxh0 krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:41.237221 15425 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39713 } }
I20260925 14:59:41.237550 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:41.217361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.4.243.254:39713)
0925 14:59:41.218132 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:41.218145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:41.218192 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:59:41.220883 (+ 2691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:41.220910 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:41.221352 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:41.221771 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:41.223156 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:41.224449 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:41.225971 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:41.226683 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:41.226860 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:41.230715 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:41.230727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:41.230740 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:41.230778 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:41.233755 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:41.234378 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:41.234386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:41.234397 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:41.234557 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:41.234961 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:41.234966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:41.234970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:41.235471 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:41.235795 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:41.236166 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 14:59:41.236574 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":327,"threads_started":1}
I20260925 14:59:41.237627 15425 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:41.237759 15425 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e60f15ef3e4af5828d5a7b4e3513a9, State: Initialized, Role: FOLLOWER
I20260925 14:59:41.238320 15425 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39713 } }
I20260925 14:59:41.238576 15425 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:41.238758 15425 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:41.238973 15425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:41.241397 15425 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39713 } }
I20260925 14:59:41.241899 15425 leader_election.cc:302] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4e60f15ef3e4af5828d5a7b4e3513a9; no voters:
I20260925 14:59:41.242519 15425 leader_election.cc:288] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:41.242803 15430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:41.243997 15430 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [term 1 LEADER]: Becoming Leader. State: Replica: c4e60f15ef3e4af5828d5a7b4e3513a9, State: Running, Role: LEADER
I20260925 14:59:41.244524 15430 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39713 } }
I20260925 14:59:41.244702 15425 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:41.247828 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39713 } } }
I20260925 14:59:41.248178 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:41.247457 15432 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e60f15ef3e4af5828d5a7b4e3513a9. Latest consensus state: current_term: 1 leader_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e60f15ef3e4af5828d5a7b4e3513a9" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39713 } } }
I20260925 14:59:41.248607 15432 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:41.258853 15440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:41.263868 15440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:41.272190 15440 catalog_manager.cc:1347] Generated new cluster ID: d358c0958e884f1ea7cc0b40de351e52
I20260925 14:59:41.272295 15440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:41.288492 15440 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:41.289742 15440 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:41.305713 15440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9: Generated new TSK 0
I20260925 14:59:41.306514 15440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 14:59:41.370003 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39713
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:41.704605 15451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:41.704847 15451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:41.704933 15451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:41.711570 15451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:41.711660 15451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:41.711784 15451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 14:59:41.721040 15451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:41.722304 15451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:41.723704 15451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:41.736474 15459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:41.736408 15457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:41.736511 15456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:41.738222 15451 server_base.cc:1034] running on GCE node
I20260925 14:59:41.738848 15451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:41.739694 15451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:41.740907 15451 hybrid_clock.cc:648] HybridClock initialized: now 1790348381740873 us; error 55 us; skew 500 ppm
I20260925 14:59:41.741195 15451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:41 dist-test-slave-qxh0 krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348381, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:41.744204 15451 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 14:59:41.745661 15451 webserver.cc:460] Webserver started at http://127.4.243.193:33581/ using document root <none> and password file <none>
I20260925 14:59:41.746349 15451 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:41.746481 15451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:41.746809 15451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:41.749599 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "41e3e7638e1748ed99a1535974740585"
format_stamp: "Formatted at 2026-09-25 14:59:41 on dist-test-slave-qxh0"
server_key: "9a1d20f0c34e07918f431a2df6429bf4"
server_key_iv: "a8e1dad9361bf07e86cf4359633ffb2e"
server_key_version: "encryptionkey@0"
I20260925 14:59:41.750301 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "41e3e7638e1748ed99a1535974740585"
format_stamp: "Formatted at 2026-09-25 14:59:41 on dist-test-slave-qxh0"
server_key: "9a1d20f0c34e07918f431a2df6429bf4"
server_key_iv: "a8e1dad9361bf07e86cf4359633ffb2e"
server_key_version: "encryptionkey@0"
I20260925 14:59:41.755303 15451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 14:59:41.758388 15466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:41.759721 15451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 14:59:41.759886 15451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "41e3e7638e1748ed99a1535974740585"
format_stamp: "Formatted at 2026-09-25 14:59:41 on dist-test-slave-qxh0"
server_key: "9a1d20f0c34e07918f431a2df6429bf4"
server_key_iv: "a8e1dad9361bf07e86cf4359633ffb2e"
server_key_version: "encryptionkey@0"
I20260925 14:59:41.760076 15451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:41.776711 15451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:41.780413 15451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:41.780696 15451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:41.781566 15451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:41.782837 15451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:41.782950 15451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:41.783106 15451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:41.783176 15451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:41.815863 15451 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:42669
I20260925 14:59:41.815910 15578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:42669 every 8 connection(s)
I20260925 14:59:41.817114 15451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:41.824044 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 15451
I20260925 14:59:41.824327 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:41.824954 5071 external_mini_cluster.cc:1428] Setting key b0370adae9642dbba5693007dc68b1de
Sep 25 14:59:41 dist-test-slave-qxh0 krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348381, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:41.837483 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:41.819619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:46473 (local address 127.4.243.254:39713)
0925 14:59:41.820025 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:41.820029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:41.821013 (+ 984us) server_negotiation.cc:407] Connection header received
0925 14:59:41.822831 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:41.822837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:41.822929 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:41.823065 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:41.824619 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:41.825473 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:41.827212 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:41.827612 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:41.827709 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:41.831788 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:41.831795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:41.831800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:41.831841 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:41.834544 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:41.835207 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:41.835211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:41.835213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:41.835353 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:41.835785 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:41.835790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:41.835793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:41.836042 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:41.836188 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:41.836887 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 14:59:41.837090 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
I20260925 14:59:41.838457 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:41.820157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39713 (local address 127.4.243.193:46473)
0925 14:59:41.820841 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:41.820892 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:41.822510 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:41.823293 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:41.823301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:41.823797 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:41.824460 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:41.824478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:41.825650 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:41.825654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:41.827071 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:41.827085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:41.827834 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:41.827843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:41.828194 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:41.828917 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:41.828955 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:41.831580 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:41.834696 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:41.834702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:41.834715 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:41.835093 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:41.835473 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:41.835479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:41.835482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:41.835669 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:41.836172 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:41.836180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:41.836502 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 14:59:41.836770 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 14:59:41.837287 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":273,"threads_started":1}
I20260925 14:59:41.840180 15579 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39713
I20260925 14:59:41.840498 15579 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:41.841413 15579 heartbeater.cc:511] Master 127.4.243.254:39713 requested a full tablet report, sending...
I20260925 14:59:41.843947 15390 ts_manager.cc:194] Registered new tserver with Master: 41e3e7638e1748ed99a1535974740585 (127.4.243.193:42669)
I20260925 14:59:41.846282 15390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:46473
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 14:59:41.877559 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39713
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:42.197968 15585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:42.198268 15585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:42.198357 15585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:42.205597 15585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:42.205704 15585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:42.205869 15585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 14:59:42.215221 15585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:42.216326 15585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:42.217671 15585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:42.230123 15591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.230140 15593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.230343 15590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:42.230829 15585 server_base.cc:1034] running on GCE node
I20260925 14:59:42.231374 15585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:42.232229 15585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:42.233474 15585 hybrid_clock.cc:648] HybridClock initialized: now 1790348382233440 us; error 47 us; skew 500 ppm
I20260925 14:59:42.233765 15585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:42 dist-test-slave-qxh0 krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:42.236650 15585 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 14:59:42.237936 15585 webserver.cc:460] Webserver started at http://127.4.243.194:35003/ using document root <none> and password file <none>
I20260925 14:59:42.238530 15585 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:42.238690 15585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:42.238945 15585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:42.241719 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "c506843fb06a4da9ad5d678606c3c263"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-qxh0"
server_key: "020ad2bdd8e1f73f456faa3b7d2ab176"
server_key_iv: "4c2db5ef6bb46df58240f8654f0df2a6"
server_key_version: "encryptionkey@0"
I20260925 14:59:42.242430 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "c506843fb06a4da9ad5d678606c3c263"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-qxh0"
server_key: "020ad2bdd8e1f73f456faa3b7d2ab176"
server_key_iv: "4c2db5ef6bb46df58240f8654f0df2a6"
server_key_version: "encryptionkey@0"
I20260925 14:59:42.247947 15585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 14:59:42.251336 15600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:42.252792 15585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 14:59:42.252948 15585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "c506843fb06a4da9ad5d678606c3c263"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-qxh0"
server_key: "020ad2bdd8e1f73f456faa3b7d2ab176"
server_key_iv: "4c2db5ef6bb46df58240f8654f0df2a6"
server_key_version: "encryptionkey@0"
I20260925 14:59:42.253140 15585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:42.282660 15585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:42.286541 15585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:42.286839 15585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:42.287758 15585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:42.289023 15585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:42.289093 15585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:42.289170 15585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:42.289217 15585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:42.320672 15585 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:32987
I20260925 14:59:42.320712 15712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:32987 every 8 connection(s)
I20260925 14:59:42.322003 15585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:42.329314 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 15585
I20260925 14:59:42.329495 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:42.329950 5071 external_mini_cluster.cc:1428] Setting key 2820f897f2cbdd156f45801157009b5c
Sep 25 14:59:42 dist-test-slave-qxh0 krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:42.341979 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.324619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:59353 (local address 127.4.243.254:39713)
0925 14:59:42.324793 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:42.324798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:42.325961 (+ 1163us) server_negotiation.cc:407] Connection header received
0925 14:59:42.327820 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:42.327826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:42.327919 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:42.328031 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:42.329459 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.330253 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.332273 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.332692 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.332790 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.336655 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:42.336660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:42.336663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:42.336691 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:42.339217 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.339868 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.339877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.339881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.339991 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.340381 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.340386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.340389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.340625 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:42.340777 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:42.341446 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 14:59:42.341648 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:59:42.342957 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.325099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39713 (local address 127.4.243.194:59353)
0925 14:59:42.325784 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:42.325820 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:42.327546 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:42.328257 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:42.328265 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:42.328742 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:42.329281 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.329297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.330397 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.330408 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.332140 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.332153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.332938 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.332948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.333267 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.333967 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:42.333998 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:42.336449 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:42.339351 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:42.339359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:42.339374 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:42.339749 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:42.340092 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:42.340097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:42.340101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:42.340268 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:42.340755 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:42.340763 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:42.341094 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 14:59:42.341332 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 14:59:42.341807 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":288,"threads_started":1}
I20260925 14:59:42.344285 15713 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39713
I20260925 14:59:42.344619 15713 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:42.345335 15713 heartbeater.cc:511] Master 127.4.243.254:39713 requested a full tablet report, sending...
I20260925 14:59:42.346849 15390 ts_manager.cc:194] Registered new tserver with Master: c506843fb06a4da9ad5d678606c3c263 (127.4.243.194:32987)
I20260925 14:59:42.348096 15390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:59353
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 14:59:42.377836 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39713
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:42.685705 15718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:42.686048 15718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:42.686237 15718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:42.693238 15718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:42.693389 15718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:42.693567 15718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 14:59:42.702896 15718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 14:59:42.704075 15718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:42.705292 15718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:42.717532 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.717504 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.717576 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:42.718230 15718 server_base.cc:1034] running on GCE node
I20260925 14:59:42.718899 15718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:42.719763 15718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:42.720994 15718 hybrid_clock.cc:648] HybridClock initialized: now 1790348382720957 us; error 48 us; skew 500 ppm
I20260925 14:59:42.721288 15718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 14:59:42 dist-test-slave-qxh0 krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:42.724520 15718 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 14:59:42.726056 15718 webserver.cc:460] Webserver started at http://127.4.243.195:36685/ using document root <none> and password file <none>
I20260925 14:59:42.726809 15718 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:42.726949 15718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:42.727236 15718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:42.729980 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "4dd2c9a8ff584bcfba4cfabd544dce82"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-qxh0"
server_key: "743634953b282f2f2f42df70455fb959"
server_key_iv: "f28ec1dfae06d7503999bceb36bd6452"
server_key_version: "encryptionkey@0"
I20260925 14:59:42.730774 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "4dd2c9a8ff584bcfba4cfabd544dce82"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-qxh0"
server_key: "743634953b282f2f2f42df70455fb959"
server_key_iv: "f28ec1dfae06d7503999bceb36bd6452"
server_key_version: "encryptionkey@0"
I20260925 14:59:42.735533 15718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 14:59:42.738500 15733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:42.739835 15718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:59:42.740013 15718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "4dd2c9a8ff584bcfba4cfabd544dce82"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-qxh0"
server_key: "743634953b282f2f2f42df70455fb959"
server_key_iv: "f28ec1dfae06d7503999bceb36bd6452"
server_key_version: "encryptionkey@0"
I20260925 14:59:42.740198 15718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:42.761121 15718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:42.766412 15718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:42.766794 15718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:42.768077 15718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:42.769635 15718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:42.769714 15718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:42.769837 15718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:42.769886 15718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:42.801316 15718 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:36641
I20260925 14:59:42.801362 15845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:36641 every 8 connection(s)
I20260925 14:59:42.802702 15718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:42.808740 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 15718
I20260925 14:59:42.808916 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:42.809387 5071 external_mini_cluster.cc:1428] Setting key 5e1c1ebf110205050568f55a6f759373
Sep 25 14:59:42 dist-test-slave-qxh0 krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 14:59:42.822515 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.805323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:48515 (local address 127.4.243.254:39713)
0925 14:59:42.805534 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:42.805538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:42.806741 (+ 1203us) server_negotiation.cc:407] Connection header received
0925 14:59:42.808664 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:42.808669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:42.808776 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:42.808894 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:42.810809 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.811783 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.813411 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.813823 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.813914 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.817207 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:42.817213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:42.817224 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:42.817267 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:42.819699 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.820383 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.820393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.820396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.820511 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.820919 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.820929 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.820933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.821181 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:42.821325 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:42.821970 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 14:59:42.822167 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 14:59:42.823561 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.805856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39713 (local address 127.4.243.195:48515)
0925 14:59:42.806562 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:42.806601 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:42.808399 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:42.809176 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:42.809184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:42.809764 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:42.810537 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.810566 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.811931 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.811936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.813288 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.813300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.813998 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.814003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.814180 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.814896 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:42.814920 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:42.817011 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:42.819850 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:42.819859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:42.819872 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:42.820257 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:42.820619 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:42.820625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:42.820628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:42.820796 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:42.821301 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:42.821308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:42.821605 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 14:59:42.821855 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 14:59:42.822313 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":252,"threads_started":1}
I20260925 14:59:42.824819 15846 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39713
I20260925 14:59:42.825152 15846 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:42.825834 15846 heartbeater.cc:511] Master 127.4.243.254:39713 requested a full tablet report, sending...
I20260925 14:59:42.827162 15389 ts_manager.cc:194] Registered new tserver with Master: 4dd2c9a8ff584bcfba4cfabd544dce82 (127.4.243.195:36641)
I20260925 14:59:42.827976 15389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:48515
I20260925 14:59:42.836277 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:42.850163 15579 heartbeater.cc:503] Master 127.4.243.254:39713 was elected leader, sending a full tablet report...
I20260925 14:59:42.853185 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.840591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.4.243.254:39713)
0925 14:59:42.840791 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:42.840800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:42.841228 (+ 428us) server_negotiation.cc:407] Connection header received
0925 14:59:42.841350 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:42.841354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:42.841409 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:42.841492 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:42.843004 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.843705 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.845025 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.845463 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.845562 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.847841 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:42.847846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:42.847849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:42.847885 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:42.850753 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.851316 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.851321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.851323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.851480 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.851876 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.851887 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.851890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.852190 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:42.852365 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:42.852628 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 14:59:42.852829 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 14:59:42.857056 15389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:42.863547 15389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:42.901787 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.874879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42669 (local address 127.0.0.1:53100)
0925 14:59:42.893584 (+ 18705us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:42.893605 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:42.893732 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:42.895807 (+ 2075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:42.895811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:42.895836 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:42.896207 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.896215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.897750 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.897756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.899403 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.899420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.900690 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.900696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.900937 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.900987 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:59:42.901140 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 14:59:42.901333 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18551,"mutex_wait_us":36,"thread_start_us":18417,"threads_started":1}
I20260925 14:59:42.901924 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.875323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.4.243.193:42669)
0925 14:59:42.895275 (+ 19952us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:42.895283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:42.895307 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:59:42.895395 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:42.895404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:42.895612 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:42.895812 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:42.896387 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.897568 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.899590 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.900488 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.900643 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.900876 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:42.901414 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 14:59:42.901588 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19806,"thread_start_us":19647,"threads_started":1}
I20260925 14:59:42.902696 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.874632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:36641 (local address 127.0.0.1:48510)
0925 14:59:42.891736 (+ 17104us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:42.891791 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:42.892023 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:42.895320 (+ 3297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:42.895328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:42.895356 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:42.895736 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.895754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.897050 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.897056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.899478 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.899492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.901186 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.901191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.901394 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.901422 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 14:59:42.901512 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:59:42.901635 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16739,"mutex_wait_us":23,"thread_start_us":15757,"threads_started":1}
I20260925 14:59:42.903136 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.875447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.4.243.195:36641)
0925 14:59:42.894616 (+ 19169us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:42.894626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:42.894671 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:59:42.894802 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:42.894812 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:42.895101 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:42.895320 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:42.895939 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.896914 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.899636 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.900987 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.901098 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.901303 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:42.901895 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 14:59:42.902056 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18907,"thread_start_us":19185,"threads_started":1}
I20260925 14:59:42.904259 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.875355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:32987 (local address 127.0.0.1:55322)
0925 14:59:42.894685 (+ 19330us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:42.894715 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:42.894816 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:42.895618 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:42.895628 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:42.895650 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:42.897392 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.897407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.899247 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.899255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.901395 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:42.901407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.903590 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.903614 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:42.903805 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.903835 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:59:42.903920 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:59:42.904022 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19162,"mutex_wait_us":1315,"thread_start_us":19019,"threads_started":1}
I20260925 14:59:42.904811 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.875943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.4.243.194:32987)
0925 14:59:42.894195 (+ 18252us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:42.894207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:42.894946 (+ 739us) server_negotiation.cc:407] Connection header received
0925 14:59:42.895072 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:42.895079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:42.895287 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:42.895465 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:42.897594 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.899115 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.901564 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.903170 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.903299 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.903460 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:42.904488 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0925 14:59:42.904599 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18108,"thread_start_us":17995,"threads_started":1}
I20260925 14:59:42.906029 15514 tablet_service.cc:1467] Processing CreateTablet for tablet cbfe08de35d24facb6528d53f868b6d1 (DEFAULT_TABLE table=test-table [id=33b17bdc177c441f97cbe9bc9b52bebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:42.907328 15514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfe08de35d24facb6528d53f868b6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:42.907682 15648 tablet_service.cc:1467] Processing CreateTablet for tablet cbfe08de35d24facb6528d53f868b6d1 (DEFAULT_TABLE table=test-table [id=33b17bdc177c441f97cbe9bc9b52bebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:42.908922 15648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfe08de35d24facb6528d53f868b6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:42.909735 15781 tablet_service.cc:1467] Processing CreateTablet for tablet cbfe08de35d24facb6528d53f868b6d1 (DEFAULT_TABLE table=test-table [id=33b17bdc177c441f97cbe9bc9b52bebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:42.910945 15781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbfe08de35d24facb6528d53f868b6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:42.917841 15864 tablet_bootstrap.cc:492] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Bootstrap starting.
I20260925 14:59:42.921670 15864 tablet_bootstrap.cc:654] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:42.922669 15865 tablet_bootstrap.cc:492] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Bootstrap starting.
I20260925 14:59:42.923079 15864 log.cc:824] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:42.926285 15865 tablet_bootstrap.cc:654] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:42.926334 15864 tablet_bootstrap.cc:492] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: No bootstrap required, opened a new log
I20260925 14:59:42.926339 15867 tablet_bootstrap.cc:492] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Bootstrap starting.
I20260925 14:59:42.926584 15864 ts_tablet_manager.cc:1397] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 14:59:42.927557 15865 log.cc:824] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:42.929414 15867 tablet_bootstrap.cc:654] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:42.930501 15865 tablet_bootstrap.cc:492] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: No bootstrap required, opened a new log
I20260925 14:59:42.930549 15867 log.cc:824] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:42.930780 15865 ts_tablet_manager.cc:1397] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260925 14:59:42.932946 15867 tablet_bootstrap.cc:492] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: No bootstrap required, opened a new log
I20260925 14:59:42.933197 15867 ts_tablet_manager.cc:1397] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 14:59:42.937003 15864 raft_consensus.cc:348] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:42.937448 15864 raft_consensus.cc:374] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:42.937542 15864 raft_consensus.cc:729] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e3e7638e1748ed99a1535974740585, State: Initialized, Role: FOLLOWER
I20260925 14:59:42.938149 15864 consensus_queue.cc:260] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:42.939800 15864 ts_tablet_manager.cc:1428] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260925 14:59:42.940508 15867 raft_consensus.cc:348] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:42.940904 15867 raft_consensus.cc:374] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:42.941025 15867 raft_consensus.cc:729] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd2c9a8ff584bcfba4cfabd544dce82, State: Initialized, Role: FOLLOWER
I20260925 14:59:42.940851 15865 raft_consensus.cc:348] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:42.941226 15865 raft_consensus.cc:374] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:42.941320 15865 raft_consensus.cc:729] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c506843fb06a4da9ad5d678606c3c263, State: Initialized, Role: FOLLOWER
I20260925 14:59:42.941588 15867 consensus_queue.cc:260] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:42.941848 15865 consensus_queue.cc:260] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:42.943918 15846 heartbeater.cc:503] Master 127.4.243.254:39713 was elected leader, sending a full tablet report...
I20260925 14:59:42.945417 15713 heartbeater.cc:503] Master 127.4.243.254:39713 was elected leader, sending a full tablet report...
I20260925 14:59:42.946156 15865 ts_tablet_manager.cc:1428] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260925 14:59:42.946542 15867 ts_tablet_manager.cc:1428] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
W20260925 14:59:43.054358 15847 tablet.cc:2369] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:43.069386 15580 tablet.cc:2369] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:43.074055 15714 tablet.cc:2369] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:59:43.202719 15871 raft_consensus.cc:482] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:43.203006 15871 raft_consensus.cc:504] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:43.204576 15871 leader_election.cc:288] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41e3e7638e1748ed99a1535974740585 (127.4.243.193:42669), c506843fb06a4da9ad5d678606c3c263 (127.4.243.194:32987)
I20260925 14:59:43.210973 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.204944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:32987 (local address 127.4.243.195:36079)
0925 14:59:43.205260 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.205282 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.205461 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.206049 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.206053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.206072 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.206353 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.206364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.207785 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.207792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.209099 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.209111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.210234 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.210240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.210433 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.210465 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.210551 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.210709 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"spinlock_wait_cycles":26368}
I20260925 14:59:43.210997 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.205040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:36079 (local address 127.4.243.194:32987)
0925 14:59:43.205254 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.205260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.205621 (+ 361us) server_negotiation.cc:407] Connection header received
0925 14:59:43.205714 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.205719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.205797 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.205881 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.206513 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.207630 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.209257 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.210078 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.210242 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.210318 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.210623 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.210752 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:59:43.212162 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.205040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:42439 (local address 127.4.243.193:42669)
0925 14:59:43.205515 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.205521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.206164 (+ 643us) server_negotiation.cc:407] Connection header received
0925 14:59:43.206323 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.206329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.206402 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.206496 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.207029 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.207714 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.209996 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.211099 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.211249 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.211321 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.211796 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.211932 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292}
I20260925 14:59:43.212224 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.205033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42669 (local address 127.4.243.195:42439)
0925 14:59:43.206026 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.206043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.206186 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.206522 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.206526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.206547 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.206861 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.206869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.207885 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.207891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.209830 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.209843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.211451 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.211456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.211622 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.211658 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.211763 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.211875 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":636,"threads_started":1}
I20260925 14:59:43.212144 15668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08de35d24facb6528d53f868b6d1" candidate_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c506843fb06a4da9ad5d678606c3c263" is_pre_election: true
I20260925 14:59:43.212656 15668 raft_consensus.cc:2454] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd2c9a8ff584bcfba4cfabd544dce82 in term 0.
I20260925 14:59:43.213182 15534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08de35d24facb6528d53f868b6d1" candidate_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e3e7638e1748ed99a1535974740585" is_pre_election: true
I20260925 14:59:43.213330 15736 leader_election.cc:302] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dd2c9a8ff584bcfba4cfabd544dce82, c506843fb06a4da9ad5d678606c3c263; no voters:
I20260925 14:59:43.213687 15871 raft_consensus.cc:2781] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:43.213639 15534 raft_consensus.cc:2454] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dd2c9a8ff584bcfba4cfabd544dce82 in term 0.
I20260925 14:59:43.213826 15871 raft_consensus.cc:482] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:43.213940 15871 raft_consensus.cc:3037] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:43.216270 15871 raft_consensus.cc:504] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:43.216905 15871 leader_election.cc:288] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [CANDIDATE]: Term 1 election: Requested vote from peers 41e3e7638e1748ed99a1535974740585 (127.4.243.193:42669), c506843fb06a4da9ad5d678606c3c263 (127.4.243.194:32987)
I20260925 14:59:43.217264 15534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08de35d24facb6528d53f868b6d1" candidate_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e3e7638e1748ed99a1535974740585"
I20260925 14:59:43.217365 15668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbfe08de35d24facb6528d53f868b6d1" candidate_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c506843fb06a4da9ad5d678606c3c263"
I20260925 14:59:43.217453 15534 raft_consensus.cc:3037] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:43.217553 15668 raft_consensus.cc:3037] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:43.219792 15534 raft_consensus.cc:2454] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd2c9a8ff584bcfba4cfabd544dce82 in term 1.
I20260925 14:59:43.220177 15737 leader_election.cc:302] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41e3e7638e1748ed99a1535974740585, 4dd2c9a8ff584bcfba4cfabd544dce82; no voters:
I20260925 14:59:43.220430 15871 raft_consensus.cc:2781] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:43.220597 15668 raft_consensus.cc:2454] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dd2c9a8ff584bcfba4cfabd544dce82 in term 1.
I20260925 14:59:43.220996 15871 raft_consensus.cc:686] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 1 LEADER]: Becoming Leader. State: Replica: 4dd2c9a8ff584bcfba4cfabd544dce82, State: Running, Role: LEADER
I20260925 14:59:43.221441 15871 consensus_queue.cc:237] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } }
I20260925 14:59:43.225700 15390 catalog_manager.cc:5495] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dd2c9a8ff584bcfba4cfabd544dce82 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dd2c9a8ff584bcfba4cfabd544dce82" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:43.256129 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.250989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.4.243.195:36641)
0925 14:59:43.251187 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.251194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.251290 (+ 96us) server_negotiation.cc:407] Connection header received
0925 14:59:43.251521 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.251525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.251607 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.251750 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:43.252122 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.252824 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.254129 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.254483 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.254618 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.255054 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:43.255544 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:43.255697 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.255756 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.255831 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:59:43.263618 15668 raft_consensus.cc:1261] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 1 FOLLOWER]: Refusing update from remote peer 4dd2c9a8ff584bcfba4cfabd544dce82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:43.263623 15534 raft_consensus.cc:1261] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 1 FOLLOWER]: Refusing update from remote peer 4dd2c9a8ff584bcfba4cfabd544dce82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:43.264546 15874 consensus_queue.cc:1035] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41e3e7638e1748ed99a1535974740585" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:43.264833 15871 consensus_queue.cc:1035] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c506843fb06a4da9ad5d678606c3c263" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:43.277310 15878 mvcc.cc:205] Tried to move back new op lower bound from 7333266977838358528 to 7333266977678770176. Current Snapshot: MvccSnapshot[applied={T|T < 7333266977838358528}]
I20260925 14:59:43.288690 15390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:43.288951 15390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:43.292308 15390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4e60f15ef3e4af5828d5a7b4e3513a9: Sending DeleteTablet for 3 replicas of tablet cbfe08de35d24facb6528d53f868b6d1
I20260925 14:59:43.293367 15781 tablet_service.cc:1514] Processing DeleteTablet for tablet cbfe08de35d24facb6528d53f868b6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:43 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:48510
I20260925 14:59:43.293358 15514 tablet_service.cc:1514] Processing DeleteTablet for tablet cbfe08de35d24facb6528d53f868b6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:43 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:53100
I20260925 14:59:43.293938 15648 tablet_service.cc:1514] Processing DeleteTablet for tablet cbfe08de35d24facb6528d53f868b6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:43 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:55322
I20260925 14:59:43.295444 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 15451
I20260925 14:59:43.306071 15887 tablet_replica.cc:330] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: stopping tablet replica
I20260925 14:59:43.306594 15887 raft_consensus.cc:2229] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:43.307204 15887 raft_consensus.cc:2258] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:43.307996 15888 tablet_replica.cc:330] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: stopping tablet replica
I20260925 14:59:43.308300 15888 raft_consensus.cc:2229] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:43.308779 15888 raft_consensus.cc:2258] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:43.310293 15887 ts_tablet_manager.cc:1905] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:43.310573 15888 ts_tablet_manager.cc:1905] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:43.316987 15887 ts_tablet_manager.cc:1918] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:43.317142 15887 log.cc:1196] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/cbfe08de35d24facb6528d53f868b6d1
I20260925 14:59:43.317384 15888 ts_tablet_manager.cc:1918] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:43.317615 15888 log.cc:1196] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/cbfe08de35d24facb6528d53f868b6d1
I20260925 14:59:43.317653 15887 ts_tablet_manager.cc:1939] T cbfe08de35d24facb6528d53f868b6d1 P 41e3e7638e1748ed99a1535974740585: Deleting consensus metadata
I20260925 14:59:43.318220 15888 ts_tablet_manager.cc:1939] T cbfe08de35d24facb6528d53f868b6d1 P 4dd2c9a8ff584bcfba4cfabd544dce82: Deleting consensus metadata
I20260925 14:59:43.319458 15377 catalog_manager.cc:4841] TS 41e3e7638e1748ed99a1535974740585 (127.4.243.193:42669): tablet cbfe08de35d24facb6528d53f868b6d1 (table test-table [id=33b17bdc177c441f97cbe9bc9b52bebc]) successfully deleted
I20260925 14:59:43.319763 15374 catalog_manager.cc:4841] TS 4dd2c9a8ff584bcfba4cfabd544dce82 (127.4.243.195:36641): tablet cbfe08de35d24facb6528d53f868b6d1 (table test-table [id=33b17bdc177c441f97cbe9bc9b52bebc]) successfully deleted
I20260925 14:59:43.323231 15889 tablet_replica.cc:330] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: stopping tablet replica
I20260925 14:59:43.323657 15889 raft_consensus.cc:2229] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:43.324133 15889 raft_consensus.cc:2258] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:43.326225 15889 ts_tablet_manager.cc:1905] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 14:59:43 dist-test-slave-qxh0 krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 14:59:43.333173 15889 ts_tablet_manager.cc:1918] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:43.333312 15889 log.cc:1196] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/cbfe08de35d24facb6528d53f868b6d1
I20260925 14:59:43.333833 15889 ts_tablet_manager.cc:1939] T cbfe08de35d24facb6528d53f868b6d1 P c506843fb06a4da9ad5d678606c3c263: Deleting consensus metadata
I20260925 14:59:43.335505 15376 catalog_manager.cc:4841] TS c506843fb06a4da9ad5d678606c3c263 (127.4.243.194:32987): tablet cbfe08de35d24facb6528d53f868b6d1 (table test-table [id=33b17bdc177c441f97cbe9bc9b52bebc]) successfully deleted
I20260925 14:59:43.336663 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.297633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.4.243.193:42669)
0925 14:59:43.297845 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.297854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.309707 (+ 11853us) server_negotiation.cc:407] Connection header received
0925 14:59:43.310194 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.310203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.310316 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.310400 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:43.319549 (+ 9149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.321060 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.323441 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.324105 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.324197 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.329631 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:43.329647 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:43.329665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:43.329706 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:43.333367 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.334182 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.334193 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.334196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.334342 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.334794 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.334799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.334803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.335212 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:43.335492 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.336045 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.336290 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:59:43.337474 15574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:43.441534 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 15451
I20260925 14:59:43.459697 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 15585
Sep 25 14:59:43 dist-test-slave-qxh0 krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 14:59:43.475893 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.460485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.4.243.194:32987)
0925 14:59:43.460689 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.460696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.460718 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:59:43.460840 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.460847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.460935 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.461051 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:43.462172 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.463302 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.464542 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.465167 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.465293 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.469867 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:43.469891 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:43.469915 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:43.469959 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:43.472859 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.473659 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.473666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.473668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.473799 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.474227 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.474231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.474233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.474853 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:43.475084 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.475424 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.475578 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:59:43.476776 15708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:43.563609 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 15585
I20260925 14:59:43.587368 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 15718
Sep 25 14:59:43 dist-test-slave-qxh0 krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 14:59:43.603639 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.588211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.4.243.195:36641)
0925 14:59:43.588429 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.588435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.588470 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:59:43.588564 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.588570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.588648 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.588766 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:43.589949 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.590851 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.592127 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.592582 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.592720 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.597484 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:43.597495 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:43.597511 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:43.597545 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:43.600744 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.601546 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.601553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.601555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.601719 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.602135 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.602138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.602140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.602510 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:43.602799 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.603119 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.603291 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 14:59:43.604403 15841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:43.721976 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 15718
I20260925 14:59:43.737039 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 15358
I20260925 14:59:43.738157 15420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 14:59:43.844473 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 15358
2026-09-25T14:59:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4362 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:43 dist-test-slave-qxh0 krb5kdc[15897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:43 dist-test-slave-qxh0 krb5kdc[15897](info): set up 2 sockets
Sep 25 14:59:43 dist-test-slave-qxh0 krb5kdc[15897](info): commencing operation
krb5kdc: starting...
W20260925 14:59:44.927597 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 14:59:45 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/rangeradmin_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/rangeradmin_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/rangerlookup_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/rangerlookup_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
I20260925 14:59:45.235544 5071 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Tl3Wl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/postgres -l logfile start
2026-09-25 14:59:47.654 UTC [15927] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:59:47.654 UTC [15927] LOG: listening on IPv4 address "127.4.243.212", port 35109
2026-09-25 14:59:47.657 UTC [15927] LOG: listening on Unix socket "/tmp/.s.PGSQL.35109"
2026-09-25 14:59:47.678 UTC [15930] LOG: database system was shut down at 2026-09-25 14:59:46 UTC
2026-09-25 14:59:47.682 UTC [15927] LOG: database system is ready to accept connections
I20260925 14:59:48.659122 5071 mini_postgres.cc:96] Postgres bound to 35109
2026-09-25 14:59:48.675 UTC [15938] FATAL: database "slave" does not exist
127.4.243.212:35109 - accepting connections
I20260925 14:59:48.676394 5071 mini_ranger.cc:161] Starting Ranger...
I20260925 14:59:48.701011 5071 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:59:49.173506 5071 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:59:49.173650 5071 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-admin
2026-09-25 14:59:49,452 [I] DB FLAVOR :POSTGRES
2026-09-25 14:59:49,453 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:59:49,453 [I] Checking connection..
2026-09-25 14:59:49,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:59:49,686 [I] Checking connection passed.
2026-09-25 14:59:49,686 [I] --------- Verifying version history table ---------
2026-09-25 14:59:49,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:49,918 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:59:49,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:50,174 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:59:50,174 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:59:50,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:59:50.423 UTC [16002] WARNING: there is no transaction in progress
2026-09-25 14:59:50,429 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:59:50,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:50,653 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:59:50,654 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:59:50,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:59:50,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:59:51,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qxh0.c.gcp-upstream.internal','N') ;"
2026-09-25 14:59:51,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:59:51,621 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:59:51,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:59:51,870 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:59:51,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:59:52,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:59:52,338 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:59:52,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:59:53.265 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.281 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.296 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.311 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.325 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.343 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.394 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.400 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.415 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.425 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.435 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.448 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.457 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.463 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.470 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:53.476 UTC [16110] WARNING: there is no transaction in progress
2026-09-25 14:59:54,095 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:59:54,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:59:54,330 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:59:54,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:59:54,578 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:59:54,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:59:54,826 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:59:54,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:59:55,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:59:55,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qxh0.c.gcp-upstream.internal';"
2026-09-25 14:59:55,559 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:59:55,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:59:55,797 [I] DB_PATCHES have already been applied
I20260925 14:59:55.802798 5071 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Tl3Wl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:59:55.802906 5071 mini_ranger.cc:193] Using host: 127.4.243.212
I20260925 14:59:55.817931 5071 mini_ranger.cc:227] Ranger admin URL: http://127.4.243.212:41437
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:59:56.920568 5071 mini_ranger.cc:160] Time spent starting Ranger: real 8.244s user 0.004s sys 0.033s
Sep 25, 2026 3:00:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:00:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:00:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:00:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:00:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:00:17.322072 5071 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/rangerkms_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/rangerkms_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.243.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.243.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:00:17 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348417, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:00:17.490819 5071 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:00:17.511940 5071 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:00:18.379516 5071 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:00:18.379680 5071 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms
I20260925 15:00:18.379884 5071 mini_ranger_kms.cc:227] Using postgres at 127.4.243.212:35109
2026-09-25 15:00:18,793 [I] DB FLAVOR :POSTGRES
2026-09-25 15:00:18,793 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:00:18,793 [I] Checking connection
2026-09-25 15:00:18,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:00:19,031 [I] connection success
2026-09-25 15:00:19,031 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:00:19,031 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:00:19,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:00:19,299 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:00:19,299 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:00:19,299 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:00:19,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:35109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:00:19,551 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:00:20.032894 5071 mini_ranger_kms.cc:310] Created kms service
I20260925 15:00:20.252822 5071 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:00:20.299624 5071 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:00:20 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348417, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:00:20.703518 5071 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:00:20.703859 5071 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:00:20.703903 5071 mini_ranger_kms.cc:240] Using host: 127.4.243.212
I20260925 15:00:20.725487 5071 mini_ranger_kms.cc:276] Ranger KMS PID: 16290
I20260925 15:00:20.725687 5071 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.243.212:43039
Sep 25, 2026 3:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:00:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:00:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:00:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:00:23.015019 5071 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.524s user 0.103s sys 0.327s
I20260925 15:00:23.015275 5071 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:00:23.015481 5071 mini_ranger_kms.cc:401] 127.4.243.212:43039/kms/v1/keys
Sep 25 15:00:23 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348423, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.243.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:00:23 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348423, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.243.212@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
Sep 25 15:00:23 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348423, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.243.212@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
Sep 25 15:00:26 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:00:26.147047 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38515
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:38515
--ranger_config_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:43039/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:26.489521 16341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:26.489784 16341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:26.489866 16341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:26.497217 16341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:26.497320 16341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:26.497370 16341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:26.497414 16341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:26.497460 16341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:26.507575 16341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.243.212:43039/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:38515
--ranger_config_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:00:26.508819 16341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:26.510046 16341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:26.521883 16346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:26.521883 16347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:26.521932 16349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:26.522722 16341 server_base.cc:1034] running on GCE node
I20260925 15:00:26.523384 16341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:26.524596 16341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:26.525820 16341 hybrid_clock.cc:648] HybridClock initialized: now 1790348426525787 us; error 46 us; skew 500 ppm
I20260925 15:00:26.526158 16341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:00:26 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:26.532706 16341 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:00:26.534111 16341 webserver.cc:460] Webserver started at http://127.4.243.254:41057/ using document root <none> and password file <none>
I20260925 15:00:26.534726 16341 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:26.534818 16341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:26.535044 16341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:26 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:00:26.585266 16341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "1e040951a56643989cf70fee6f299f38"
format_stamp: "Formatted at 2026-09-25 15:00:26 on dist-test-slave-qxh0"
server_key: "9a8d00f99f99a712fc0eee4c6c30a8f5"
server_key_iv: "ab906a97249a91788985caead7e5f0c9"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:26.586112 16341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "1e040951a56643989cf70fee6f299f38"
format_stamp: "Formatted at 2026-09-25 15:00:26 on dist-test-slave-qxh0"
server_key: "9a8d00f99f99a712fc0eee4c6c30a8f5"
server_key_iv: "ab906a97249a91788985caead7e5f0c9"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:26.591492 16341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:00:26.607151 16356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:26.608681 16341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:00:26.608873 16341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "1e040951a56643989cf70fee6f299f38"
format_stamp: "Formatted at 2026-09-25 15:00:26 on dist-test-slave-qxh0"
server_key: "9a8d00f99f99a712fc0eee4c6c30a8f5"
server_key_iv: "ab906a97249a91788985caead7e5f0c9"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:26.609068 16341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:26.632251 16341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:26.636075 16341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:26.636369 16341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:26.656232 16341 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:38515
I20260925 15:00:26.656260 16407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:38515 every 8 connection(s)
I20260925 15:00:26.657503 16341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:26.661103 16408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:26.663489 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 16341
I20260925 15:00:26.663682 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:00:26.664343 5071 external_mini_cluster.cc:1428] Setting key b0a72ad3b5b38d38d624c466461a82df
I20260925 15:00:26.674300 16408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38: Bootstrap starting.
I20260925 15:00:26.677599 16408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:26.678550 16408 log.cc:824] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:00:26 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:00:26.681564 16408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38: No bootstrap required, opened a new log
I20260925 15:00:26.688908 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:26.666441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.4.243.254:38515)
0925 15:00:26.667667 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:26.667683 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:26.667740 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:00:26.669936 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:26.669962 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:26.670455 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:26.670912 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:26.672041 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:26.673889 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:26.675291 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:26.676347 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:26.676515 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:26.680534 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:26.680544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:26.680561 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:26.680599 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:26.684742 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:26.685363 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:26.685371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:26.685381 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:26.685557 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:26.685899 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:26.685904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:26.685907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:26.686573 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:26.686868 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:26.687259 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 15:00:26.687640 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":374,"threads_started":1}
I20260925 15:00:26.692065 16408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e040951a56643989cf70fee6f299f38" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38515 } }
I20260925 15:00:26.692361 16408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:26.692426 16408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e040951a56643989cf70fee6f299f38, State: Initialized, Role: FOLLOWER
I20260925 15:00:26.692855 16408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e040951a56643989cf70fee6f299f38" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38515 } }
I20260925 15:00:26.693044 16408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:26.693130 16408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:26.693251 16408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:26.695204 16408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e040951a56643989cf70fee6f299f38" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38515 } }
I20260925 15:00:26.695607 16408 leader_election.cc:302] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e040951a56643989cf70fee6f299f38; no voters:
I20260925 15:00:26.696096 16408 leader_election.cc:288] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:00:26.696235 16413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:26.697657 16413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [term 1 LEADER]: Becoming Leader. State: Replica: 1e040951a56643989cf70fee6f299f38, State: Running, Role: LEADER
I20260925 15:00:26.698275 16413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e040951a56643989cf70fee6f299f38" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38515 } }
I20260925 15:00:26.698716 16408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:26.700821 16415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e040951a56643989cf70fee6f299f38. Latest consensus state: current_term: 1 leader_uuid: "1e040951a56643989cf70fee6f299f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e040951a56643989cf70fee6f299f38" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38515 } } }
I20260925 15:00:26.701202 16415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:26.701747 16414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e040951a56643989cf70fee6f299f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e040951a56643989cf70fee6f299f38" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38515 } } }
I20260925 15:00:26.702028 16414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:26.706312 16408 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:00:26.707217 16421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:26.718616 16421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:26.729877 16421 catalog_manager.cc:1347] Generated new cluster ID: 062d3ef274e341f29f4a1a2d7c1c411e
I20260925 15:00:26.729984 16421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:00:26.745824 16421 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:00:26.747929 16421 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:26.754805 16421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38: Generated new TSK 0
I20260925 15:00:26.755774 16421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 15:00:27.510762 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:43039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38515
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:00:27 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348427, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:00:27.943614 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:27.944015 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:27.944232 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:27.955576 16453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:27.955768 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:27.955986 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:00:27.967764 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.243.212:43039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:00:27.969508 16453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:27.971208 16453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:27.991460 16463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:27.992049 16464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:27.995745 16467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:27.996572 16453 server_base.cc:1034] running on GCE node
I20260925 15:00:27.997303 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:27.998548 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:27.999814 16453 hybrid_clock.cc:648] HybridClock initialized: now 1790348427999740 us; error 59 us; skew 500 ppm
I20260925 15:00:28.000694 16453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:28.004679 16453 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:00:28.006294 16453 webserver.cc:460] Webserver started at http://127.4.243.193:42983/ using document root <none> and password file <none>
I20260925 15:00:28.007086 16453 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:28.007228 16453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:28.007550 16453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:00:28.028730 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-qxh0"
server_key: "626ac79cf774071c58216fedaaf30e96"
server_key_iv: "2240638efe934ddb98b815e5a1f1f923"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.035432 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-qxh0"
server_key: "626ac79cf774071c58216fedaaf30e96"
server_key_iv: "2240638efe934ddb98b815e5a1f1f923"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.052788 16453 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.008s sys 0.007s
I20260925 15:00:28.079885 16473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:28.081813 16453 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260925 15:00:28.082062 16453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-qxh0"
server_key: "626ac79cf774071c58216fedaaf30e96"
server_key_iv: "2240638efe934ddb98b815e5a1f1f923"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.082386 16453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:28.130546 16453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:28.135658 16453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:28.136049 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:28.143117 16453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:28.148806 16453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:28.148968 16453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:28.149147 16453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:28.149253 16453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348427, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:00:28.288512 16453 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:34609
I20260925 15:00:28.288553 16585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:34609 every 8 connection(s)
I20260925 15:00:28.290074 16453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:00:28.293318 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 16453
I20260925 15:00:28.293510 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:00:28.294303 5071 external_mini_cluster.cc:1428] Setting key 4840edb6dd5e2d36720b45c780d924bc
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 15:00:28.330799 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:28.305813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:50713 (local address 127.4.243.254:38515)
0925 15:00:28.312599 (+ 6786us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:28.312605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:28.312627 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:00:28.313626 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:28.313631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:28.313719 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:28.313810 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:28.315825 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.316913 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.319164 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.319723 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.319830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:28.323919 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:28.323926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:28.323931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:28.323967 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:28.327342 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:28.328185 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:28.328191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:28.328194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:28.328324 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:28.328804 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:28.328809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:28.328812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:28.329141 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:28.329266 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:28.330255 (+ 989us) server_negotiation.cc:299] Negotiation successful
0925 15:00:28.330480 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6659,"thread_start_us":8757,"threads_started":1}
I20260925 15:00:28.332031 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:28.306507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38515 (local address 127.4.243.193:50713)
0925 15:00:28.310754 (+ 4247us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:28.310813 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:28.313221 (+ 2408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:28.314270 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:28.314282 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:28.314889 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:28.315630 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:28.315651 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.317092 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.317097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:28.318989 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:28.319012 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.320102 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.320108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:28.320361 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:28.321117 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:28.321144 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:28.323674 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:28.327497 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:28.327506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:28.327524 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:28.328051 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:28.328445 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:28.328450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:28.328453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:28.328673 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:28.329342 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:28.329350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:28.329737 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 15:00:28.330082 (+ 345us) client_negotiation.cc:240] Negotiation successful
0925 15:00:28.330570 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3829,"thread_start_us":3318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 15:00:28.333652 16586 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38515
I20260925 15:00:28.333982 16586 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:28.334741 16586 heartbeater.cc:511] Master 127.4.243.254:38515 requested a full tablet report, sending...
I20260925 15:00:28.336812 16373 ts_manager.cc:194] Registered new tserver with Master: 8be819e6b41d4c2b8d2d4a93dc8536bf (127.4.243.193:34609)
I20260925 15:00:28.338220 16373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:50713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 15:00:28.362383 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:43039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38515
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348427, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
W20260925 15:00:28.826434 16592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:28.826911 16592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:28.827099 16592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:28.834498 16592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:28.834654 16592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:28.834841 16592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:00:28.844661 16592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.243.212:43039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:00:28.845893 16592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:28.847230 16592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:28.860085 16606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:28.860162 16607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:28.860123 16609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:28.861215 16592 server_base.cc:1034] running on GCE node
I20260925 15:00:28.861925 16592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:28.863395 16592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:28.864625 16592 hybrid_clock.cc:648] HybridClock initialized: now 1790348428864583 us; error 40 us; skew 500 ppm
I20260925 15:00:28.864936 16592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:28.868211 16592 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:00:28.869617 16592 webserver.cc:460] Webserver started at http://127.4.243.194:44153/ using document root <none> and password file <none>
I20260925 15:00:28.870289 16592 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:28.870453 16592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:28.870814 16592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:00:28.883948 16592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "db4608d05ad649db864ef43c536a8fe4"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-qxh0"
server_key: "55ce765426090cbdc552b2aa9837ef9f"
server_key_iv: "df01663107916aec2d6ce2b7f4bb55d0"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.884739 16592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "db4608d05ad649db864ef43c536a8fe4"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-qxh0"
server_key: "55ce765426090cbdc552b2aa9837ef9f"
server_key_iv: "df01663107916aec2d6ce2b7f4bb55d0"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.890036 16592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:00:28.906066 16616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:28.907776 16592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:00:28.907995 16592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "db4608d05ad649db864ef43c536a8fe4"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-qxh0"
server_key: "55ce765426090cbdc552b2aa9837ef9f"
server_key_iv: "df01663107916aec2d6ce2b7f4bb55d0"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.908228 16592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:28.922444 16592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:28.926144 16592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:28.926466 16592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:28.927409 16592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:28.928725 16592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:28.928834 16592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:28.928973 16592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:28.929055 16592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:28.968607 16592 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:42635
I20260925 15:00:28.968626 16728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:42635 every 8 connection(s)
I20260925 15:00:28.969880 16592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:00:28.970837 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 16592
I20260925 15:00:28.971009 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:00:28.971496 5071 external_mini_cluster.cc:1428] Setting key 7fe45c7e0c232697ef789880b21dc5b5
Sep 25 15:00:28 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 15:00:29.004628 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:28.974653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:50627 (local address 127.4.243.254:38515)
0925 15:00:28.987136 (+ 12483us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:28.987145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:28.990395 (+ 3250us) server_negotiation.cc:407] Connection header received
0925 15:00:28.990452 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:28.990456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:28.990529 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:28.990687 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:28.992182 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.992910 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.994435 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.994925 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.995054 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:28.998304 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:28.998311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:28.998314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:28.998343 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:29.001211 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.002081 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.002086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.002091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.002241 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.002751 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.002757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.002767 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.003103 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:29.003295 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.004009 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.004220 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12302,"thread_start_us":12170,"threads_started":1}
I20260925 15:00:29.005676 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:28.973946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38515 (local address 127.4.243.194:50627)
0925 15:00:28.987773 (+ 13827us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:28.987838 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:28.990072 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:28.990954 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:28.990963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:28.991406 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:28.992002 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:28.992020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.993065 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.993070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:28.994290 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:28.994308 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.995170 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.995177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:28.995432 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:28.996111 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:28.996134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:28.998096 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:29.001392 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.001402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.001421 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.001955 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.002356 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.002361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.002364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.002556 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.003239 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:29.003246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:29.003609 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.003879 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.004440 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13510,"thread_start_us":13272,"threads_started":1}
I20260925 15:00:29.007347 16729 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38515
I20260925 15:00:29.007659 16729 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:29.008373 16729 heartbeater.cc:511] Master 127.4.243.254:38515 requested a full tablet report, sending...
I20260925 15:00:29.009792 16373 ts_manager.cc:194] Registered new tserver with Master: db4608d05ad649db864ef43c536a8fe4 (127.4.243.194:42635)
I20260925 15:00:29.011011 16373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:50627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 15:00:29.037636 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:43039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38515
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:29.341540 16586 heartbeater.cc:503] Master 127.4.243.254:38515 was elected leader, sending a full tablet report...
W20260925 15:00:29.385099 16735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:29.385447 16735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:29.385632 16735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:29.393461 16735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:29.393620 16735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:29.393805 16735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:00:29.403663 16735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.243.212:43039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:00:29.404991 16735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:29.406278 16735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:29.420572 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.420537 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.420606 16740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:29.421295 16735 server_base.cc:1034] running on GCE node
I20260925 15:00:29.421957 16735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:29.423062 16735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:29.424326 16735 hybrid_clock.cc:648] HybridClock initialized: now 1790348429424278 us; error 51 us; skew 500 ppm
I20260925 15:00:29.424679 16735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:00:29 dist-test-slave-qxh0 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:29.428092 16735 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:00:29.429783 16735 webserver.cc:460] Webserver started at http://127.4.243.195:33363/ using document root <none> and password file <none>
I20260925 15:00:29.430485 16735 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:29.430656 16735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:29.430969 16735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:29 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:00:29.445097 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "702960c8a21c4055adbc02a9bdb00974"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-qxh0"
server_key: "90400e8e70b6bd7662f05499f06ec23f"
server_key_iv: "1b799816f2db5b3bf0d8fbd0ea21bfdb"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.446170 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "702960c8a21c4055adbc02a9bdb00974"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-qxh0"
server_key: "90400e8e70b6bd7662f05499f06ec23f"
server_key_iv: "1b799816f2db5b3bf0d8fbd0ea21bfdb"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.452894 16735 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260925 15:00:29.467666 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.469435 16735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:00:29.469587 16735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "702960c8a21c4055adbc02a9bdb00974"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-qxh0"
server_key: "90400e8e70b6bd7662f05499f06ec23f"
server_key_iv: "1b799816f2db5b3bf0d8fbd0ea21bfdb"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.469780 16735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:29.493127 16735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:29.496701 16735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:29.496950 16735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:29.497783 16735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:29.499159 16735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:29.499682 16735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.499818 16735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:29.499869 16735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.534838 16735 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:34657
I20260925 15:00:29.535349 16862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:34657 every 8 connection(s)
I20260925 15:00:29.536443 16735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:00:29.539981 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 16735
I20260925 15:00:29.540208 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:00:29.540822 5071 external_mini_cluster.cc:1428] Setting key ba6a24a45a9c975c48da7eb3da44e815
Sep 25 15:00:29 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:00:29.557104 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.539131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:43321 (local address 127.4.243.254:38515)
0925 15:00:29.539623 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.539630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.540771 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 15:00:29.542862 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.542867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.542993 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.543110 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:29.544667 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.545375 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.547066 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.547506 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.547641 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.550930 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:29.550935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:29.550938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:29.550967 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:29.553887 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.554718 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.554729 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.554733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.554850 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.555317 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.555320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.555322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.555708 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:29.555841 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.556675 (+ 834us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.556854 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260925 15:00:29.558010 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.539752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38515 (local address 127.4.243.195:43321)
0925 15:00:29.540573 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.540636 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.542531 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.543379 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.543388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.543861 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:29.544507 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.544523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.545562 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.545566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.546928 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.546940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.547731 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.547741 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.547992 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.548692 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:29.548718 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:29.550711 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:29.554032 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.554043 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.554062 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.554573 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.554972 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.554977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.554980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.555160 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.555849 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:29.555858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:29.556238 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.556581 (+ 343us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.556960 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":320,"threads_started":1}
I20260925 15:00:29.559376 16863 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38515
I20260925 15:00:29.559675 16863 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:29.560375 16863 heartbeater.cc:511] Master 127.4.243.254:38515 requested a full tablet report, sending...
I20260925 15:00:29.561805 16373 ts_manager.cc:194] Registered new tserver with Master: 702960c8a21c4055adbc02a9bdb00974 (127.4.243.195:34657)
I20260925 15:00:29.562654 16373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:43321
I20260925 15:00:29.568431 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:00:29.585063 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.572799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.4.243.254:38515)
0925 15:00:29.572968 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.572972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.573423 (+ 451us) server_negotiation.cc:407] Connection header received
0925 15:00:29.573602 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.573606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.573674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.573747 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:29.575069 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.575775 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.577085 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.577473 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.577567 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.580238 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:29.580243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:29.580246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:29.580282 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:29.582691 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.583306 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.583310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.583312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.583508 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.583857 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.583860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.583862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.584115 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:29.584299 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.584653 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.584834 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:00:29.589545 16373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:00:29.596306 16373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:00:29.630144 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.608465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:34609 (local address 127.0.0.1:42120)
0925 15:00:29.609354 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.609407 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.609625 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.622307 (+ 12682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.622314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.622350 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:29.623020 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.623034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.625079 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.625088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.627458 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.627470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.629085 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.629093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.629306 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.629359 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.629470 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.629655 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":177,"thread_start_us":609,"threads_started":1}
I20260925 15:00:29.630792 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.610021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42635 (local address 127.0.0.1:33894)
0925 15:00:29.621364 (+ 11343us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.621381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.621503 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.622246 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.622253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.622282 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:29.622993 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.623013 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.625344 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.625364 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.628122 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.628136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.629909 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.629916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.630093 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.630124 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.630216 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.630356 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11217,"mutex_wait_us":201,"thread_start_us":11088,"threads_started":1}
I20260925 15:00:29.631173 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.608469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:34657 (local address 127.0.0.1:39674)
0925 15:00:29.619541 (+ 11072us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.619568 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.619732 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.622419 (+ 2687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.622425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.622485 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:29.623322 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.623337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.625167 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.625176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.628379 (+ 3203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.628388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.630349 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.630355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.630520 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.630553 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.630662 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.630799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10907,"mutex_wait_us":603,"spinlock_wait_cycles":8064,"thread_start_us":11202,"threads_started":1}
I20260925 15:00:29.631446 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.608894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.4.243.195:34657)
0925 15:00:29.620654 (+ 11760us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.620667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.620713 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:00:29.620862 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.620874 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.622203 (+ 1329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.622650 (+ 447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:29.623526 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.625022 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.628712 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.630135 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.630257 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.630471 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.630956 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.631122 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11538,"thread_start_us":11354,"threads_started":1}
I20260925 15:00:29.631829 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.608676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.4.243.193:34609)
0925 15:00:29.621692 (+ 13016us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.621698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.621722 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:00:29.621810 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.621815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.622071 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.622719 (+ 648us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:29.623225 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.624917 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.627619 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.628852 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.630691 (+ 1839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.631111 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.631190 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.631425 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12860,"thread_start_us":12714,"threads_started":1}
I20260925 15:00:29.632293 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.610574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.4.243.194:42635)
0925 15:00:29.620907 (+ 10333us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.620920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.621633 (+ 713us) server_negotiation.cc:407] Connection header received
0925 15:00:29.621748 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.621756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.622035 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.622245 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:29.623272 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.625185 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.628287 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.629689 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.629842 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.630053 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.631839 (+ 1786us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.632000 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10128,"thread_start_us":10014,"threads_started":1}
I20260925 15:00:29.635778 16798 tablet_service.cc:1467] Processing CreateTablet for tablet fae9d304a7834f708f7f17fc6dbd709e (DEFAULT_TABLE table=test-table [id=9795c80f26854ce99f4d571c04294b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:29.635576 16521 tablet_service.cc:1467] Processing CreateTablet for tablet fae9d304a7834f708f7f17fc6dbd709e (DEFAULT_TABLE table=test-table [id=9795c80f26854ce99f4d571c04294b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:29.636148 16664 tablet_service.cc:1467] Processing CreateTablet for tablet fae9d304a7834f708f7f17fc6dbd709e (DEFAULT_TABLE table=test-table [id=9795c80f26854ce99f4d571c04294b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:29.637239 16521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae9d304a7834f708f7f17fc6dbd709e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:29.637310 16798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae9d304a7834f708f7f17fc6dbd709e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:29.637310 16664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae9d304a7834f708f7f17fc6dbd709e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:29.648051 16882 tablet_bootstrap.cc:492] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Bootstrap starting.
I20260925 15:00:29.651551 16883 tablet_bootstrap.cc:492] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Bootstrap starting.
I20260925 15:00:29.652678 16882 tablet_bootstrap.cc:654] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:29.654388 16882 log.cc:824] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:29.655841 16883 tablet_bootstrap.cc:654] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:29.656944 16883 log.cc:824] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:29.657552 16882 tablet_bootstrap.cc:492] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: No bootstrap required, opened a new log
I20260925 15:00:29.655827 16884 tablet_bootstrap.cc:492] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Bootstrap starting.
I20260925 15:00:29.657845 16882 ts_tablet_manager.cc:1397] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260925 15:00:29.661242 16884 tablet_bootstrap.cc:654] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:29.662412 16884 log.cc:824] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:29.664991 16883 tablet_bootstrap.cc:492] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: No bootstrap required, opened a new log
I20260925 15:00:29.665269 16883 ts_tablet_manager.cc:1397] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.007s
I20260925 15:00:29.666522 16882 raft_consensus.cc:348] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.666965 16882 raft_consensus.cc:374] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:29.667094 16882 raft_consensus.cc:729] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be819e6b41d4c2b8d2d4a93dc8536bf, State: Initialized, Role: FOLLOWER
I20260925 15:00:29.667740 16882 consensus_queue.cc:260] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.669760 16882 ts_tablet_manager.cc:1428] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260925 15:00:29.676509 16884 tablet_bootstrap.cc:492] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: No bootstrap required, opened a new log
I20260925 15:00:29.676796 16884 ts_tablet_manager.cc:1397] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260925 15:00:29.679729 16883 raft_consensus.cc:348] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.680116 16883 raft_consensus.cc:374] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:29.680230 16883 raft_consensus.cc:729] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702960c8a21c4055adbc02a9bdb00974, State: Initialized, Role: FOLLOWER
I20260925 15:00:29.681241 16883 consensus_queue.cc:260] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.682337 16863 heartbeater.cc:503] Master 127.4.243.254:38515 was elected leader, sending a full tablet report...
I20260925 15:00:29.682956 16883 ts_tablet_manager.cc:1428] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260925 15:00:29.685158 16884 raft_consensus.cc:348] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.685497 16884 raft_consensus.cc:374] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:29.685601 16884 raft_consensus.cc:729] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4608d05ad649db864ef43c536a8fe4, State: Initialized, Role: FOLLOWER
I20260925 15:00:29.686232 16884 consensus_queue.cc:260] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.687346 16729 heartbeater.cc:503] Master 127.4.243.254:38515 was elected leader, sending a full tablet report...
I20260925 15:00:29.687865 16884 ts_tablet_manager.cc:1428] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
W20260925 15:00:29.735320 16730 tablet.cc:2369] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:00:29.788057 16864 tablet.cc:2369] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:00:29.811036 16587 tablet.cc:2369] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:00:29.829041 16890 raft_consensus.cc:482] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:29.829406 16890 raft_consensus.cc:504] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.831092 16890 leader_election.cc:288] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8be819e6b41d4c2b8d2d4a93dc8536bf (127.4.243.193:34609), 702960c8a21c4055adbc02a9bdb00974 (127.4.243.195:34657)
I20260925 15:00:29.840574 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.831505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:40451 (local address 127.4.243.195:34657)
0925 15:00:29.831701 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.831707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.832900 (+ 1193us) server_negotiation.cc:407] Connection header received
0925 15:00:29.833096 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.833101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.833177 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.833278 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:29.834032 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.835103 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.838073 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.839293 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.839443 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.839511 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.840216 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.840335 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:00:29.840579 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.831591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:41585 (local address 127.4.243.193:34609)
0925 15:00:29.831794 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.831803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.832899 (+ 1096us) server_negotiation.cc:407] Connection header received
0925 15:00:29.833096 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.833101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.833177 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.833278 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:29.834032 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.835120 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.838078 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.839302 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.839442 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.839515 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.840216 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.840335 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:00:29.840798 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.831376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:34657 (local address 127.4.243.194:40451)
0925 15:00:29.832741 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.832762 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.832960 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.833324 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.833327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.833366 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:29.833880 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.833893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.835253 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.835259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.837912 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.837927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.839574 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.839578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.839953 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.840084 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.840179 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.840345 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":305,"spinlock_wait_cycles":87808,"thread_start_us":988,"threads_started":1}
I20260925 15:00:29.841154 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.831513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:34609 (local address 127.4.243.194:41585)
0925 15:00:29.832791 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.832806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.832943 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.833318 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.833323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.833356 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:29.833890 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.833898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.835254 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.835259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.837912 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.837927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.839571 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.839577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.839949 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.840084 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.840179 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.840339 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":395,"thread_start_us":853,"threads_started":1}
I20260925 15:00:29.841872 16818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9d304a7834f708f7f17fc6dbd709e" candidate_uuid: "db4608d05ad649db864ef43c536a8fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702960c8a21c4055adbc02a9bdb00974" is_pre_election: true
I20260925 15:00:29.842191 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9d304a7834f708f7f17fc6dbd709e" candidate_uuid: "db4608d05ad649db864ef43c536a8fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" is_pre_election: true
I20260925 15:00:29.842329 16818 raft_consensus.cc:2454] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db4608d05ad649db864ef43c536a8fe4 in term 0.
I20260925 15:00:29.842684 16541 raft_consensus.cc:2454] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db4608d05ad649db864ef43c536a8fe4 in term 0.
I20260925 15:00:29.843001 16618 leader_election.cc:302] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 702960c8a21c4055adbc02a9bdb00974, db4608d05ad649db864ef43c536a8fe4; no voters:
I20260925 15:00:29.843400 16890 raft_consensus.cc:2781] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:00:29.843545 16890 raft_consensus.cc:482] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:00:29.843641 16890 raft_consensus.cc:3037] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:29.846158 16890 raft_consensus.cc:504] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.846786 16890 leader_election.cc:288] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [CANDIDATE]: Term 1 election: Requested vote from peers 8be819e6b41d4c2b8d2d4a93dc8536bf (127.4.243.193:34609), 702960c8a21c4055adbc02a9bdb00974 (127.4.243.195:34657)
I20260925 15:00:29.847092 16541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9d304a7834f708f7f17fc6dbd709e" candidate_uuid: "db4608d05ad649db864ef43c536a8fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf"
I20260925 15:00:29.847285 16541 raft_consensus.cc:3037] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:29.847216 16818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9d304a7834f708f7f17fc6dbd709e" candidate_uuid: "db4608d05ad649db864ef43c536a8fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702960c8a21c4055adbc02a9bdb00974"
I20260925 15:00:29.847455 16818 raft_consensus.cc:3037] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:29.849380 16541 raft_consensus.cc:2454] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db4608d05ad649db864ef43c536a8fe4 in term 1.
I20260925 15:00:29.849437 16818 raft_consensus.cc:2454] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db4608d05ad649db864ef43c536a8fe4 in term 1.
I20260925 15:00:29.849789 16620 leader_election.cc:302] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be819e6b41d4c2b8d2d4a93dc8536bf, db4608d05ad649db864ef43c536a8fe4; no voters:
I20260925 15:00:29.850057 16890 raft_consensus.cc:2781] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:29.850507 16890 raft_consensus.cc:686] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 1 LEADER]: Becoming Leader. State: Replica: db4608d05ad649db864ef43c536a8fe4, State: Running, Role: LEADER
I20260925 15:00:29.850945 16890 consensus_queue.cc:237] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } }
I20260925 15:00:29.855057 16373 catalog_manager.cc:5495] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to db4608d05ad649db864ef43c536a8fe4 (127.4.243.194). New cstate: current_term: 1 leader_uuid: "db4608d05ad649db864ef43c536a8fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db4608d05ad649db864ef43c536a8fe4" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42635 } health_report { overall_health: HEALTHY } } }
I20260925 15:00:29.871193 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.865498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.4.243.194:42635)
0925 15:00:29.865696 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.865703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.865798 (+ 95us) server_negotiation.cc:407] Connection header received
0925 15:00:29.866060 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.866068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.866152 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.866317 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:00:29.866792 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.867509 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.868938 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.869357 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.869490 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.870028 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:00:29.870522 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:00:29.870714 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.870796 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.870889 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:00:29.878980 16818 raft_consensus.cc:1261] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 1 FOLLOWER]: Refusing update from remote peer db4608d05ad649db864ef43c536a8fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:29.879019 16541 raft_consensus.cc:1261] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 1 FOLLOWER]: Refusing update from remote peer db4608d05ad649db864ef43c536a8fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:29.879935 16893 consensus_queue.cc:1035] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8be819e6b41d4c2b8d2d4a93dc8536bf" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:29.880242 16890 consensus_queue.cc:1035] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "702960c8a21c4055adbc02a9bdb00974" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:29.891438 16896 mvcc.cc:205] Tried to move back new op lower bound from 7333267168775172096 to 7333267168672702464. Current Snapshot: MvccSnapshot[applied={T|T < 7333267168775172096}]
I20260925 15:00:29.909266 16372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:29.909606 16372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:29.913506 16372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e040951a56643989cf70fee6f299f38: Sending DeleteTablet for 3 replicas of tablet fae9d304a7834f708f7f17fc6dbd709e
I20260925 15:00:29.914696 16798 tablet_service.cc:1514] Processing DeleteTablet for tablet fae9d304a7834f708f7f17fc6dbd709e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:29 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:39674
I20260925 15:00:29.915189 16664 tablet_service.cc:1514] Processing DeleteTablet for tablet fae9d304a7834f708f7f17fc6dbd709e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:29 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:33894
I20260925 15:00:29.915426 16521 tablet_service.cc:1514] Processing DeleteTablet for tablet fae9d304a7834f708f7f17fc6dbd709e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:29 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:42120
I20260925 15:00:29.915750 16906 tablet_replica.cc:330] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: stopping tablet replica
I20260925 15:00:29.916219 16906 raft_consensus.cc:2229] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:29.916792 16906 raft_consensus.cc:2258] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:29.917608 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 16453
I20260925 15:00:29.919070 16906 ts_tablet_manager.cc:1905] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:29.926671 16906 ts_tablet_manager.cc:1918] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:29.926824 16906 log.cc:1196] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/fae9d304a7834f708f7f17fc6dbd709e
I20260925 15:00:29.927310 16906 ts_tablet_manager.cc:1939] T fae9d304a7834f708f7f17fc6dbd709e P 702960c8a21c4055adbc02a9bdb00974: Deleting consensus metadata
I20260925 15:00:29.929157 16358 catalog_manager.cc:4841] TS 702960c8a21c4055adbc02a9bdb00974 (127.4.243.195:34657): tablet fae9d304a7834f708f7f17fc6dbd709e (table test-table [id=9795c80f26854ce99f4d571c04294b9a]) successfully deleted
I20260925 15:00:29.935813 16907 tablet_replica.cc:330] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: stopping tablet replica
I20260925 15:00:29.936658 16907 raft_consensus.cc:2229] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:00:29.937337 16907 raft_consensus.cc:2258] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:29.938167 16908 tablet_replica.cc:330] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: stopping tablet replica
I20260925 15:00:29.938556 16908 raft_consensus.cc:2229] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:29.939074 16908 raft_consensus.cc:2258] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:29.941299 16908 ts_tablet_manager.cc:1905] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:29.941303 16907 ts_tablet_manager.cc:1905] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:00:29 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 15:00:29.948869 16907 ts_tablet_manager.cc:1918] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:29.948870 16908 ts_tablet_manager.cc:1918] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:29.949055 16908 log.cc:1196] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/fae9d304a7834f708f7f17fc6dbd709e
I20260925 15:00:29.949540 16907 log.cc:1196] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/fae9d304a7834f708f7f17fc6dbd709e
I20260925 15:00:29.949630 16908 ts_tablet_manager.cc:1939] T fae9d304a7834f708f7f17fc6dbd709e P 8be819e6b41d4c2b8d2d4a93dc8536bf: Deleting consensus metadata
I20260925 15:00:29.950080 16907 ts_tablet_manager.cc:1939] T fae9d304a7834f708f7f17fc6dbd709e P db4608d05ad649db864ef43c536a8fe4: Deleting consensus metadata
I20260925 15:00:29.951565 16360 catalog_manager.cc:4841] TS 8be819e6b41d4c2b8d2d4a93dc8536bf (127.4.243.193:34609): tablet fae9d304a7834f708f7f17fc6dbd709e (table test-table [id=9795c80f26854ce99f4d571c04294b9a]) successfully deleted
I20260925 15:00:29.951848 16360 catalog_manager.cc:4841] TS db4608d05ad649db864ef43c536a8fe4 (127.4.243.194:42635): tablet fae9d304a7834f708f7f17fc6dbd709e (table test-table [id=9795c80f26854ce99f4d571c04294b9a]) successfully deleted
I20260925 15:00:29.954770 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.927082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.4.243.193:34609)
0925 15:00:29.927279 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.927287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.935121 (+ 7834us) server_negotiation.cc:407] Connection header received
0925 15:00:29.935323 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.935331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.935430 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.935531 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:29.937375 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.938561 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.940588 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.941223 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.941340 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.946994 (+ 5654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:29.947010 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:29.947029 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:29.947081 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:29.951202 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.952636 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.952646 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.952649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.952788 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.953176 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.953183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.953186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.953547 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:29.953836 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.954132 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.954363 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:00:29.955384 16581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:00:30.075626 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 16453
I20260925 15:00:30.093096 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 16592
Sep 25 15:00:30 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:00:30.110901 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:30.093877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.4.243.194:42635)
0925 15:00:30.094058 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:30.094064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:30.094085 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:00:30.094253 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:30.094259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:30.094345 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:30.094465 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:30.095870 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.097115 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.098502 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.099349 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.099474 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:30.103930 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:30.103950 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:30.103974 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:30.104021 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:30.107956 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.108703 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.108712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.108716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.108887 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.109233 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.109237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.109241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.109736 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:30.110064 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:30.110229 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 15:00:30.110461 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:00:30.111657 16724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:00:30.232597 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 16592
I20260925 15:00:30.250753 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 16735
Sep 25 15:00:30 dist-test-slave-qxh0 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 15:00:30.264865 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:30.251528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.4.243.195:34657)
0925 15:00:30.251691 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:30.251696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:30.251713 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:00:30.251822 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:30.251826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:30.251886 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:30.251969 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:30.253210 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.253919 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.255303 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.255675 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.255765 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:30.259896 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:30.259909 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:30.259925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:30.259960 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:30.262469 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.263130 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.263138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.263140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.263263 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.263605 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.263608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.263610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.263965 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:30.264170 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:30.264439 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:00:30.264599 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:00:30.265592 16858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:00:30.354584 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 16735
I20260925 15:00:30.370983 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 16341
I20260925 15:00:30.372117 16403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:00:30.484313 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 16341
I20260925 15:00:30.508738 5071 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:00:30.844193 5071 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:00:30.844398 5071 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:00:31.190200 5071 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:00:31.190 UTC [15927] LOG: received smart shutdown request
2026-09-25 15:00:31.194 UTC [15927] LOG: background worker "logical replication launcher" (PID 15936) exited with exit code 1
2026-09-25 15:00:31.194 UTC [15931] LOG: shutting down
2026-09-25 15:00:31.371 UTC [15927] LOG: database system is shut down
2026-09-25T15:00:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47649 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:00:31 dist-test-slave-qxh0 krb5kdc[16922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:00:31 dist-test-slave-qxh0 krb5kdc[16922](info): set up 2 sockets
Sep 25 15:00:31 dist-test-slave-qxh0 krb5kdc[16922](info): commencing operation
krb5kdc: starting...
W20260925 15:00:32.594946 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:00:32 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:00:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/rangeradmin_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/rangeradmin_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/rangerlookup_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/rangerlookup_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
I20260925 15:00:32.913089 5071 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Tl3Wl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/postgres -l logfile start
2026-09-25 15:00:35.402 UTC [16952] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:00:35.402 UTC [16952] LOG: listening on IPv4 address "127.4.243.212", port 47567
2026-09-25 15:00:35.406 UTC [16952] LOG: listening on Unix socket "/tmp/.s.PGSQL.47567"
2026-09-25 15:00:35.429 UTC [16955] LOG: database system was shut down at 2026-09-25 15:00:33 UTC
2026-09-25 15:00:35.434 UTC [16952] LOG: database system is ready to accept connections
I20260925 15:00:36.406544 5071 mini_postgres.cc:96] Postgres bound to 47567
2026-09-25 15:00:36.423 UTC [16963] FATAL: database "slave" does not exist
127.4.243.212:47567 - accepting connections
I20260925 15:00:36.423909 5071 mini_ranger.cc:161] Starting Ranger...
I20260925 15:00:36.450073 5071 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:00:36.933193 5071 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:00:36.933288 5071 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-admin
2026-09-25 15:00:37,219 [I] DB FLAVOR :POSTGRES
2026-09-25 15:00:37,219 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:00:37,219 [I] Checking connection..
2026-09-25 15:00:37,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:00:37,480 [I] Checking connection passed.
2026-09-25 15:00:37,481 [I] --------- Verifying version history table ---------
2026-09-25 15:00:37,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:00:37,717 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:00:37,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:00:37,961 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:00:37,961 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:00:37,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:00:38.192 UTC [17027] WARNING: there is no transaction in progress
2026-09-25 15:00:38,199 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:00:38,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:00:38,457 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:00:38,457 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:00:38,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:00:38,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:00:38,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qxh0.c.gcp-upstream.internal','N') ;"
2026-09-25 15:00:39,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:00:39,477 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:00:39,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:00:39,736 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:00:39,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:00:39,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:00:40,194 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:00:40,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:00:41.151 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.167 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.182 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.198 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.213 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.230 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.283 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.290 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.304 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.314 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.324 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.338 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.347 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.354 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.361 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41.367 UTC [17135] WARNING: there is no transaction in progress
2026-09-25 15:00:41,997 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:00:41,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:00:42,252 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:00:42,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:00:42,489 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:00:42,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:00:42,714 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:00:42,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:00:42,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:00:43,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qxh0.c.gcp-upstream.internal';"
2026-09-25 15:00:43,445 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:00:43,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:00:43,676 [I] DB_PATCHES have already been applied
I20260925 15:00:43.682308 5071 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Tl3Wl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:00:43.682399 5071 mini_ranger.cc:193] Using host: 127.4.243.212
I20260925 15:00:43.696242 5071 mini_ranger.cc:227] Ranger admin URL: http://127.4.243.212:34605
Sep 25, 2026 3:00:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:00:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:00:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:00:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:00:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:00:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:00:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:00:44.821933 5071 mini_ranger.cc:160] Time spent starting Ranger: real 8.398s user 0.003s sys 0.034s
Sep 25, 2026 3:01:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:01:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:01:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:01:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:01:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:01:06.220538 5071 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.243.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.243.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/rangerkms_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.243.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/rangerkms_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.243.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.243.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.243.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/HTTP_127.4.243.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:01:06 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348466, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:01:06.378973 5071 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:01:06.398777 5071 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:01:07.254060 5071 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:01:07.254171 5071 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms
I20260925 15:01:07.254216 5071 mini_ranger_kms.cc:227] Using postgres at 127.4.243.212:47567
2026-09-25 15:01:07,603 [I] DB FLAVOR :POSTGRES
2026-09-25 15:01:07,604 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:01:07,604 [I] Checking connection
2026-09-25 15:01:07,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:01:07,857 [I] connection success
2026-09-25 15:01:07,857 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:01:07,857 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:01:07,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:01:08,092 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:01:08,093 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:01:08,093 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:01:08,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.243.212:47567/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:01:08,372 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:01:08.827883 5071 mini_ranger_kms.cc:310] Created kms service
I20260925 15:01:09.019585 5071 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:01:09.063107 5071 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:01:09 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348466, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:09.474820 5071 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:01:09.475157 5071 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Tl3Wl/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Tl3Wl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:01:09.475235 5071 mini_ranger_kms.cc:240] Using host: 127.4.243.212
I20260925 15:01:09.489914 5071 mini_ranger_kms.cc:276] Ranger KMS PID: 17315
I20260925 15:01:09.490051 5071 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.243.212:58713
Sep 25, 2026 3:01:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:01:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:01:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:01:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:01:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:01:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 15:01:11 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348471, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.243.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:11.707000 5071 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.328s user 0.060s sys 0.298s
I20260925 15:01:11.707259 5071 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:01:11.707428 5071 mini_ranger_kms.cc:401] 127.4.243.212:58713/kms/v1/keys
Sep 25 15:01:12 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348471, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.243.212@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
Sep 25 15:01:12 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348471, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.243.212@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
Sep 25 15:01:14 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:01:14.602887 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35227
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35227
--ranger_config_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:14.927670 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:14.927927 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:14.928009 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:14.934795 17366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:14.934880 17366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:14.934933 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:14.935042 17366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:14.935111 17366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:14.944453 17366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.243.212:58713/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35227
--ranger_config_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:14.945673 17366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:14.946981 17366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:14.958997 17372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:14.959093 17374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:14.959656 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:14.959861 17366 server_base.cc:1034] running on GCE node
I20260925 15:01:14.960758 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:14.962230 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:14.963431 17366 hybrid_clock.cc:648] HybridClock initialized: now 1790348474963386 us; error 36 us; skew 500 ppm
I20260925 15:01:14.963816 17366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:14 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348474, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:14.973075 17366 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:14.974754 17366 webserver.cc:460] Webserver started at http://127.4.243.254:36093/ using document root <none> and password file <none>
I20260925 15:01:14.975449 17366 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:14.975589 17366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:14.975896 17366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:01:14 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348474, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:15.025426 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "f259776446434c6c94e15019ea94848b"
format_stamp: "Formatted at 2026-09-25 15:01:15 on dist-test-slave-qxh0"
server_key: "13daf4b8e5549f7dd0f5bc1faba77eb3"
server_key_iv: "5f0a26561984d7e59676444bbe73a109"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:15.026571 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "f259776446434c6c94e15019ea94848b"
format_stamp: "Formatted at 2026-09-25 15:01:15 on dist-test-slave-qxh0"
server_key: "13daf4b8e5549f7dd0f5bc1faba77eb3"
server_key_iv: "5f0a26561984d7e59676444bbe73a109"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:15.033630 17366 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260925 15:01:15.054018 17381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:15.055761 17366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:01:15.055976 17366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "f259776446434c6c94e15019ea94848b"
format_stamp: "Formatted at 2026-09-25 15:01:15 on dist-test-slave-qxh0"
server_key: "13daf4b8e5549f7dd0f5bc1faba77eb3"
server_key_iv: "5f0a26561984d7e59676444bbe73a109"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:15.056241 17366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:15.087885 17366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:15.091359 17366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:15.091604 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:15.109061 17366 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:35227
I20260925 15:01:15.109084 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:35227 every 8 connection(s)
I20260925 15:01:15.110327 17366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:15.113895 17433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:15.115733 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 17366
I20260925 15:01:15.115942 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:01:15.116720 5071 external_mini_cluster.cc:1428] Setting key 39f0de92cf7eb557fadf9635818d5499
I20260925 15:01:15.126470 17433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Bootstrap starting.
I20260925 15:01:15.130312 17433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Neither blocks nor log segments found. Creating new log.
Sep 25 15:01:15 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:15.131485 17433 log.cc:824] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:15.134896 17433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: No bootstrap required, opened a new log
I20260925 15:01:15.140684 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:15.118828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.4.243.254:35227)
0925 15:01:15.119689 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:15.119700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:15.119745 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:01:15.121418 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:15.121442 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:15.121808 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:15.122130 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:15.123858 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:15.125183 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:15.127076 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:15.127743 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:15.127878 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:15.132634 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:15.132641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:15.132653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:15.132687 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:15.136452 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:15.137087 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:15.137096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:15.137104 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:15.137308 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:15.137641 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:15.137647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:15.137654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:15.138340 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:15.138656 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:15.138991 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:01:15.139384 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":437,"threads_started":1}
I20260925 15:01:15.144141 17433 raft_consensus.cc:348] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:15.144435 17433 raft_consensus.cc:374] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:15.144505 17433 raft_consensus.cc:729] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f259776446434c6c94e15019ea94848b, State: Initialized, Role: FOLLOWER
I20260925 15:01:15.144941 17433 consensus_queue.cc:260] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:15.145129 17433 raft_consensus.cc:388] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:15.145217 17433 raft_consensus.cc:482] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:15.145331 17433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:15.147544 17433 raft_consensus.cc:504] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:15.147939 17433 leader_election.cc:302] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f259776446434c6c94e15019ea94848b; no voters:
I20260925 15:01:15.148485 17433 leader_election.cc:288] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:15.148663 17438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:15.149338 17438 raft_consensus.cc:686] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 LEADER]: Becoming Leader. State: Replica: f259776446434c6c94e15019ea94848b, State: Running, Role: LEADER
I20260925 15:01:15.149894 17438 consensus_queue.cc:237] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:15.150413 17433 sys_catalog.cc:564] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:15.152632 17440 sys_catalog.cc:455] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f259776446434c6c94e15019ea94848b. Latest consensus state: current_term: 1 leader_uuid: "f259776446434c6c94e15019ea94848b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } } }
I20260925 15:01:15.153007 17440 sys_catalog.cc:458] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:15.152874 17439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f259776446434c6c94e15019ea94848b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } } }
I20260925 15:01:15.153234 17439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:15.155421 17445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:15.160885 17445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:15.161651 17433 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:01:15.191254 17445 catalog_manager.cc:1347] Generated new cluster ID: 1e8ec05a40e24d778f82956c1b8cd029
I20260925 15:01:15.191399 17445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:15.212482 17445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:15.214389 17445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:15.223524 17445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Generated new TSK 0
I20260925 15:01:15.224588 17445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 15:01:15.885984 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:35227
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:01:16 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:01:16.270128 17478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:16.270490 17478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:16.270678 17478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:16.278919 17478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:16.279085 17478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:16.279263 17478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:01:16.290830 17478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.243.212:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:16.292202 17478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:16.293602 17478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:16.310194 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.310951 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.313283 17492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:16.314075 17478 server_base.cc:1034] running on GCE node
I20260925 15:01:16.314839 17478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:16.316177 17478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:16.317451 17478 hybrid_clock.cc:648] HybridClock initialized: now 1790348476317348 us; error 85 us; skew 500 ppm
I20260925 15:01:16.317842 17478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:16 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:16.321224 17478 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:01:16.323943 17478 webserver.cc:460] Webserver started at http://127.4.243.193:34933/ using document root <none> and password file <none>
I20260925 15:01:16.324693 17478 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:16.324842 17478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:16.325275 17478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:01:16 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:16.344300 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "4c1fd57ac31342a8b0b13051a0b7499c"
format_stamp: "Formatted at 2026-09-25 15:01:16 on dist-test-slave-qxh0"
server_key: "03236b742f38ee6c0451e0bd642b1f8f"
server_key_iv: "9f5cf0caf073bdebce8a277ec2a6514e"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:16.345360 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "4c1fd57ac31342a8b0b13051a0b7499c"
format_stamp: "Formatted at 2026-09-25 15:01:16 on dist-test-slave-qxh0"
server_key: "03236b742f38ee6c0451e0bd642b1f8f"
server_key_iv: "9f5cf0caf073bdebce8a277ec2a6514e"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:16.350915 17478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:01:16.373528 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.375604 17478 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260925 15:01:16.375875 17478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "4c1fd57ac31342a8b0b13051a0b7499c"
format_stamp: "Formatted at 2026-09-25 15:01:16 on dist-test-slave-qxh0"
server_key: "03236b742f38ee6c0451e0bd642b1f8f"
server_key_iv: "9f5cf0caf073bdebce8a277ec2a6514e"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:16.376209 17478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:16.458109 17478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:16.463220 17478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:16.463625 17478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:16.471474 17478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:16.473176 17478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:16.473285 17478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.473438 17478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:16.473519 17478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.535953 17478 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:44945
I20260925 15:01:16.535969 17610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:44945 every 8 connection(s)
I20260925 15:01:16.537997 17478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:16.539649 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 17478
I20260925 15:01:16.539839 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:16.540668 5071 external_mini_cluster.cc:1428] Setting key 2909415e0512c4462e7bca974e0135a5
Sep 25 15:01:16 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 15:01:16.589960 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.544405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:33763 (local address 127.4.243.254:35227)
0925 15:01:16.566129 (+ 21724us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:16.566135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:16.566159 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:01:16.567883 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:16.567890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:16.568026 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:16.568175 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:16.570351 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.571538 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.574005 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.574654 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.574752 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.580322 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:16.580329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:16.580334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:16.580384 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:16.584191 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.585086 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.585092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.585096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.585362 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.585945 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.585950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.585954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.586381 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:16.586511 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:16.589367 (+ 2856us) server_negotiation.cc:299] Negotiation successful
0925 15:01:16.589587 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21596,"thread_start_us":21751,"threads_started":1}
I20260925 15:01:16.590668 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.543799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35227 (local address 127.4.243.193:33763)
0925 15:01:16.564642 (+ 20843us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:16.564717 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:16.567419 (+ 2702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:16.568640 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:16.568653 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:16.569278 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:16.570112 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.570137 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.571729 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.571735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.573831 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.573846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.575097 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.575105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.575385 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.576293 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:16.576338 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:16.580069 (+ 3731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:16.584345 (+ 4276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.584355 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.584373 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.584960 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.585522 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.585528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.585532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.585795 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.586677 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:16.586686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:16.587124 (+ 438us) client_negotiation.cc:769] Sending connection context
0925 15:01:16.587439 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 15:01:16.587966 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20478,"thread_start_us":20235,"threads_started":1}
I20260925 15:01:16.592741 17611 heartbeater.cc:347] Connected to a master server at 127.4.243.254:35227
I20260925 15:01:16.593056 17611 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:16.593921 17611 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
I20260925 15:01:16.596364 17398 ts_manager.cc:194] Registered new tserver with Master: 4c1fd57ac31342a8b0b13051a0b7499c (127.4.243.193:44945)
I20260925 15:01:16.598377 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:33763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 15:01:16.626260 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:35227
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:01:16 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
Sep 25 15:01:17 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
W20260925 15:01:17.072369 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:17.072677 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:17.072857 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:17.081226 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:17.081411 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:17.081614 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:01:17.095887 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.243.212:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:17.097613 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:17.099408 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:17.120607 17624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:17.122684 17625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:17.126129 17627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:17.128901 17617 server_base.cc:1034] running on GCE node
I20260925 15:01:17.129913 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:17.132582 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:17.133895 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790348477133794 us; error 89 us; skew 500 ppm
I20260925 15:01:17.134299 17617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:17 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:17.138382 17617 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:01:17.159087 17617 webserver.cc:460] Webserver started at http://127.4.243.194:38235/ using document root <none> and password file <none>
I20260925 15:01:17.160130 17617 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:17.160321 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:17.160775 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:01:17 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:17.183661 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "d771f639e51f499fab0225e07ca24ac7"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "49e8160747fbf5795e1b1453350b2b94"
server_key_iv: "b878518e325f4029685f0665e4d52dee"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:17.184873 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "d771f639e51f499fab0225e07ca24ac7"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "49e8160747fbf5795e1b1453350b2b94"
server_key_iv: "b878518e325f4029685f0665e4d52dee"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:17.204874 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.007s sys 0.011s
I20260925 15:01:17.223218 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.226995 17617 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260925 15:01:17.227185 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "d771f639e51f499fab0225e07ca24ac7"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "49e8160747fbf5795e1b1453350b2b94"
server_key_iv: "b878518e325f4029685f0665e4d52dee"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:17.227434 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:17.286377 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:17.292052 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:17.292412 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:17.299749 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:17.308147 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:17.308291 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.308465 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:17.308569 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.460392 17617 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:36679
I20260925 15:01:17.462073 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:17.460412 17753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:36679 every 8 connection(s)
I20260925 15:01:17.464254 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 17617
I20260925 15:01:17.464437 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:17.464959 5071 external_mini_cluster.cc:1428] Setting key 63c23c2d6dd1df5374313e791f2101be
Sep 25 15:01:17 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:17.487002 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.465028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:57065 (local address 127.4.243.254:35227)
0925 15:01:17.469815 (+ 4787us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.469822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.469838 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:01:17.469888 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.469896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.469978 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.470118 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:17.471696 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.472438 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.473974 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.474393 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.474530 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.479045 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:17.479058 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:17.479062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:17.479094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:17.482961 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.483955 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.483965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.483971 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.484142 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.484700 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.484707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.484714 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.485117 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:17.485270 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.486188 (+ 918us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.486456 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4667,"thread_start_us":4552,"threads_started":1}
I20260925 15:01:17.487973 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.465870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35227 (local address 127.4.243.194:57065)
0925 15:01:17.466886 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.466935 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:17.469261 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:17.470371 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:17.470381 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:17.470940 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:17.471530 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.471548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.472592 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.472597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.473829 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.473840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.474663 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.474672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.474885 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.475566 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:17.475588 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:17.478815 (+ 3227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:17.483144 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:17.483154 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:17.483173 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:17.483801 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:17.484270 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:17.484275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:17.484279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:17.484568 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:17.485266 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:17.485275 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:17.485720 (+ 445us) client_negotiation.cc:769] Sending connection context
0925 15:01:17.486044 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 15:01:17.486626 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":596,"threads_started":1}
I20260925 15:01:17.489523 17754 heartbeater.cc:347] Connected to a master server at 127.4.243.254:35227
I20260925 15:01:17.489845 17754 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:17.490682 17754 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
I20260925 15:01:17.492372 17398 ts_manager.cc:194] Registered new tserver with Master: d771f639e51f499fab0225e07ca24ac7 (127.4.243.194:36679)
I20260925 15:01:17.493438 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:57065
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 15:01:17.523317 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:35227
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:17.602617 17611 heartbeater.cc:503] Master 127.4.243.254:35227 was elected leader, sending a full tablet report...
W20260925 15:01:17.840595 17760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:17.840935 17760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:17.841107 17760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:17.848237 17760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:17.848384 17760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:17.848563 17760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:01:17.858014 17760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.243.212:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:17.859508 17760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:17.860868 17760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:17.873313 17765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:17.873378 17766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:17.873345 17768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:17.874040 17760 server_base.cc:1034] running on GCE node
I20260925 15:01:17.874523 17760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:17.875391 17760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:17.876598 17760 hybrid_clock.cc:648] HybridClock initialized: now 1790348477876553 us; error 35 us; skew 500 ppm
I20260925 15:01:17.876888 17760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:17 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:17.879837 17760 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:01:17.882316 17760 webserver.cc:460] Webserver started at http://127.4.243.195:34731/ using document root <none> and password file <none>
I20260925 15:01:17.883046 17760 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:17.883162 17760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:17.883417 17760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:01:17 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:17.897267 17760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "ba018b4110634fb58384a5b49530669c"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "a95ef4df1183ce801f6e9fef3829d780"
server_key_iv: "d5c026a9c6219d12f76ddac76a68e887"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:17.898075 17760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "ba018b4110634fb58384a5b49530669c"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "a95ef4df1183ce801f6e9fef3829d780"
server_key_iv: "d5c026a9c6219d12f76ddac76a68e887"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:17.903297 17760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:01:17.923355 17775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.926755 17760 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.006s
I20260925 15:01:17.926971 17760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "ba018b4110634fb58384a5b49530669c"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "a95ef4df1183ce801f6e9fef3829d780"
server_key_iv: "d5c026a9c6219d12f76ddac76a68e887"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:17.927254 17760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:17.948385 17760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:17.952138 17760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:17.952426 17760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:17.953347 17760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:17.954679 17760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:17.954783 17760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.954926 17760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:17.955004 17760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.987499 17760 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42071
I20260925 15:01:17.987515 17887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42071 every 8 connection(s)
I20260925 15:01:17.988799 17760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:17.997967 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 17760
I20260925 15:01:17.998162 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:17.998744 5071 external_mini_cluster.cc:1428] Setting key 8374def53ba9e4aa3544b5c51203fdaa
Sep 25 15:01:17 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:18.008281 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.991284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:41167 (local address 127.4.243.254:35227)
0925 15:01:17.991752 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.991756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.992744 (+ 988us) server_negotiation.cc:407] Connection header received
0925 15:01:17.994399 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.994405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.994494 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.994634 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:17.995968 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.996674 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.998218 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.998913 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.999058 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:18.002555 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:18.002560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:18.002564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:18.002646 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:18.005354 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.006087 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.006094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.006097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.006227 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.006664 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.006672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.006674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.006932 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:18.007092 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:18.007728 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:01:18.007942 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":231,"threads_started":1}
I20260925 15:01:18.009176 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.991863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35227 (local address 127.4.243.195:41167)
0925 15:01:17.992593 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.992629 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:17.994144 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:17.994857 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:17.994866 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:17.995251 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:17.995801 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.995816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.996832 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.996842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.998052 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.998063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.999121 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.999126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.999321 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:18.000042 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:18.000075 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:18.002337 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:18.005514 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:18.005523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:18.005538 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:18.005969 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:18.006343 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:18.006348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:18.006351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:18.006521 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:18.007053 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:18.007073 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:18.007402 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 15:01:18.007649 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 15:01:18.008136 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260925 15:01:18.010416 17888 heartbeater.cc:347] Connected to a master server at 127.4.243.254:35227
I20260925 15:01:18.010763 17888 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:18.011451 17888 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
I20260925 15:01:18.012848 17398 ts_manager.cc:194] Registered new tserver with Master: ba018b4110634fb58384a5b49530669c (127.4.243.195:42071)
I20260925 15:01:18.013638 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:41167
I20260925 15:01:18.014083 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:18.016332 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 17478
Sep 25 15:01:18 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 15:01:18.031910 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:18.017191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.4.243.193:44945)
0925 15:01:18.017705 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:18.017712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:18.017733 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:01:18.017781 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:18.017786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:18.017948 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:18.018079 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:18.019660 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:18.020627 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:18.022076 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:18.022567 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:18.022732 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:18.026537 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:18.026546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:18.026558 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:18.026734 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:18.029375 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.029995 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.030001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.030003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.030125 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.030405 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.030408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.030410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.030803 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:18.031012 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:18.031309 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 15:01:18.031564 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260925 15:01:18.033106 17606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:18.115494 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 17478
I20260925 15:01:18.131063 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 17617
Sep 25 15:01:18 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:01:18.146893 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:18.132013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.4.243.194:36679)
0925 15:01:18.132587 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:18.132593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:18.132617 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:01:18.132674 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:18.132680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:18.132960 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:18.133084 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:18.134429 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:18.135538 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:18.136763 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:18.137265 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:18.137355 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:18.141429 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:18.141439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:18.141451 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:18.141487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:18.144315 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.144965 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.144971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.144973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.145102 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.145411 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.145414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.145416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.145710 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:18.145980 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:18.146185 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 15:01:18.146369 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":312,"threads_started":1}
I20260925 15:01:18.148089 17749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:18.230294 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 17617
I20260925 15:01:18.245806 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 17760
Sep 25 15:01:18 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 15:01:18.262046 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:18.246927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.4.243.195:42071)
0925 15:01:18.247579 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:18.247586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:18.247614 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:01:18.247672 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:18.247678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:18.247891 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:18.248021 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:18.249267 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:18.250219 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:18.251628 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:18.252188 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:18.252325 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:18.256627 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:18.256637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:18.256651 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:18.256679 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:18.259356 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.260003 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.260010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.260012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.260128 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:18.260484 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:18.260492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:18.260494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:18.260808 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:18.261119 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:18.261458 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 15:01:18.261701 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":332,"threads_started":1}
I20260925 15:01:18.263278 17883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:18.343979 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 17760
I20260925 15:01:18.359210 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 17366
I20260925 15:01:18.360352 17428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:18.464612 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 17366
I20260925 15:01:18.485931 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35227
--webserver_interface=127.4.243.254
--webserver_port=36093
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35227
--ranger_config_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:18.795373 17900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:18.795698 17900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:18.795862 17900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:18.803119 17900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:18.803270 17900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:18.803375 17900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:18.803483 17900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:18.803563 17900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:18.813573 17900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35227
--ranger_config_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=36093
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:18.814931 17900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:18.816176 17900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:18.827716 17905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:18.827761 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:18.827802 17906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:18.828763 17900 server_base.cc:1034] running on GCE node
I20260925 15:01:18.829442 17900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:18.830717 17900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:18.831940 17900 hybrid_clock.cc:648] HybridClock initialized: now 1790348478831896 us; error 63 us; skew 500 ppm
I20260925 15:01:18.832278 17900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:18 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:18.835276 17900 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:18.836550 17900 webserver.cc:460] Webserver started at http://127.4.243.254:36093/ using document root <none> and password file <none>
I20260925 15:01:18.837147 17900 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:18.837286 17900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:01:18 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:18.894867 17900 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:01:18.898018 17916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:18.899509 17900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:18.899685 17900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "f259776446434c6c94e15019ea94848b"
format_stamp: "Formatted at 2026-09-25 15:01:15 on dist-test-slave-qxh0"
server_key: "13daf4b8e5549f7dd0f5bc1faba77eb3"
server_key_iv: "5f0a26561984d7e59676444bbe73a109"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:18.900588 17900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:18.930747 17900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:18.934792 17900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:18.935068 17900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:18.952917 17900 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:35227
I20260925 15:01:18.952934 17967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:35227 every 8 connection(s)
I20260925 15:01:18.954310 17900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:18.956570 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 17900
I20260925 15:01:18.957549 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:44945
--local_ip_for_outbound_sockets=127.4.243.193
--tserver_master_addrs=127.4.243.254:35227
--webserver_port=34933
--webserver_interface=127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:18.963856 17968 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:01:18.967797 17968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Bootstrap starting.
I20260925 15:01:18.986305 17968 log.cc:824] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:18.994248 17968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:01:18.994788 17968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Bootstrap complete.
I20260925 15:01:19.003086 17968 raft_consensus.cc:348] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:19.004076 17968 raft_consensus.cc:729] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f259776446434c6c94e15019ea94848b, State: Initialized, Role: FOLLOWER
I20260925 15:01:19.004781 17968 consensus_queue.cc:260] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:19.005030 17968 raft_consensus.cc:388] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:19.005143 17968 raft_consensus.cc:482] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:19.005358 17968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:01:19.009094 17968 raft_consensus.cc:504] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:19.009550 17968 leader_election.cc:302] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f259776446434c6c94e15019ea94848b; no voters:
I20260925 15:01:19.010188 17968 leader_election.cc:288] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:01:19.010315 17972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:01:19.011497 17972 raft_consensus.cc:686] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [term 2 LEADER]: Becoming Leader. State: Replica: f259776446434c6c94e15019ea94848b, State: Running, Role: LEADER
I20260925 15:01:19.012025 17972 consensus_queue.cc:237] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } }
I20260925 15:01:19.012312 17968 sys_catalog.cc:564] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:19.014339 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f259776446434c6c94e15019ea94848b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } } }
I20260925 15:01:19.014748 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:19.014956 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f259776446434c6c94e15019ea94848b. Latest consensus state: current_term: 2 leader_uuid: "f259776446434c6c94e15019ea94848b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f259776446434c6c94e15019ea94848b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35227 } } }
I20260925 15:01:19.015393 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:19.020087 17977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:19.026283 17977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:19.027773 17977 catalog_manager.cc:1259] Loaded cluster ID: 1e8ec05a40e24d778f82956c1b8cd029
I20260925 15:01:19.027910 17977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:19.031674 17977 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:19.033039 17977 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Loaded TSK: 0
I20260925 15:01:19.034348 17977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:01:19.043648 17968 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 15:01:19.342998 17970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:19.343327 17970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:19.343479 17970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:19.354403 17970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:19.354645 17970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:19.354882 17970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:01:19.367581 17970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:44945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=34933
--tserver_master_addrs=127.4.243.254:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:19.369177 17970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:19.370927 17970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:19.397893 18005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:19.398320 18006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:19.403658 18008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:19.405380 17970 server_base.cc:1034] running on GCE node
I20260925 15:01:19.406303 17970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:19.410894 17970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:19.412218 17970 hybrid_clock.cc:648] HybridClock initialized: now 1790348479412148 us; error 57 us; skew 500 ppm
I20260925 15:01:19.412609 17970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:19 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348479, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:19.421795 17970 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:01:19.439518 17970 webserver.cc:460] Webserver started at http://127.4.243.193:34933/ using document root <none> and password file <none>
I20260925 15:01:19.440770 17970 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:19.440958 17970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:01:19 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348479, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:19.481045 17970 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260925 15:01:19.499321 18016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:19.501595 17970 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.012s
I20260925 15:01:19.501950 17970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "4c1fd57ac31342a8b0b13051a0b7499c"
format_stamp: "Formatted at 2026-09-25 15:01:16 on dist-test-slave-qxh0"
server_key: "03236b742f38ee6c0451e0bd642b1f8f"
server_key_iv: "9f5cf0caf073bdebce8a277ec2a6514e"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:19.503832 17970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:19.584312 17970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:19.589303 17970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:19.589670 17970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:19.590819 17970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:19.601048 17970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:19.601202 17970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:19.601330 17970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:19.601426 17970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:19.689810 17970 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:44945
I20260925 15:01:19.689853 18129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:44945 every 8 connection(s)
I20260925 15:01:19.691376 17970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:19.697865 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 17970
I20260925 15:01:19.698657 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:36679
--local_ip_for_outbound_sockets=127.4.243.194
--tserver_master_addrs=127.4.243.254:35227
--webserver_port=38235
--webserver_interface=127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:01:19 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348479, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:19.728027 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:19.695345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35227 (local address 127.4.243.193:41837)
0925 15:01:19.697668 (+ 2323us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:19.697714 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:19.700167 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:19.709554 (+ 9387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:19.709611 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:19.710246 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:19.711135 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:19.711159 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:19.713405 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:19.713410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:19.715288 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:19.715303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:19.716462 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:19.716469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:19.716726 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:19.717502 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:19.717524 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:19.719496 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:19.723399 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:19.723408 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:19.723423 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:19.723828 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:19.724554 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:19.724559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:19.724563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:19.724739 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:19.725464 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:19.725471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:19.725796 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 15:01:19.726084 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 15:01:19.726679 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":1820,"threads_started":1}
I20260925 15:01:19.728022 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:19.696650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:41837 (local address 127.4.243.254:35227)
0925 15:01:19.707354 (+ 10704us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:19.707374 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:19.707427 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:01:19.707909 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:19.707930 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:19.708447 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:19.708856 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:19.711366 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:19.713209 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:19.715475 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:19.716225 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:19.716727 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:19.719801 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:19.719812 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:19.719831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:19.719883 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:19.723253 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:19.724269 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:19.724275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:19.724280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:19.724429 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:19.724859 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:19.724868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:19.724870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:19.725329 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:19.725541 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:19.726317 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 15:01:19.726697 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":296,"threads_started":1}
I20260925 15:01:19.731505 18130 heartbeater.cc:347] Connected to a master server at 127.4.243.254:35227
I20260925 15:01:19.731838 18130 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:19.732616 18130 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
W20260925 15:01:19.735592 18130 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:35227 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:01:19.735790 18130 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:19.736034 18130 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
I20260925 15:01:19.737152 18130 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:19.737406 18130 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
W20260925 15:01:19.738355 18130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:01:20.112026 18134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:20.112286 18134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:20.112422 18134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:20.119765 18134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:20.119865 18134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:20.120035 18134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:01:20.131397 18134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:36679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=38235
--tserver_master_addrs=127.4.243.254:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:20.133131 18134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:20.134824 18134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:20.154696 18149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:20.154983 18148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:20.158052 18151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:20.158447 18134 server_base.cc:1034] running on GCE node
I20260925 15:01:20.159207 18134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:20.162137 18134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:20.163398 18134 hybrid_clock.cc:648] HybridClock initialized: now 1790348480163322 us; error 61 us; skew 500 ppm
I20260925 15:01:20.163792 18134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:20 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:20.172019 18134 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:01:20.174054 18134 webserver.cc:460] Webserver started at http://127.4.243.194:38235/ using document root <none> and password file <none>
I20260925 15:01:20.175038 18134 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:20.175202 18134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:01:20 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:20.207636 18134 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.007s sys 0.000s
I20260925 15:01:20.219421 18159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:20.222180 18134 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.008s
I20260925 15:01:20.222420 18134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "d771f639e51f499fab0225e07ca24ac7"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "49e8160747fbf5795e1b1453350b2b94"
server_key_iv: "b878518e325f4029685f0665e4d52dee"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:20.223874 18134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:20.273607 18134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:20.278733 18134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:20.279016 18134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:20.280241 18134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:20.282023 18134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:20.282123 18134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:20.282241 18134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:20.282311 18134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:01:20 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:20.358031 18134 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:36679
I20260925 15:01:20.358109 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:36679 every 8 connection(s)
I20260925 15:01:20.359756 18134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:20.361387 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18134
I20260925 15:01:20.362114 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.243.212:58713/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:42071
--local_ip_for_outbound_sockets=127.4.243.195
--tserver_master_addrs=127.4.243.254:35227
--webserver_port=34731
--webserver_interface=127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:01:20 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:20.416731 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:20.372652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35227 (local address 127.4.243.194:35647)
0925 15:01:20.395681 (+ 23029us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:20.395730 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:20.397456 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:20.398201 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:20.398209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:20.398714 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:20.399367 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:20.399391 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:20.400889 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:20.400895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:20.403077 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:20.403100 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:20.404419 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:20.404427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:20.404701 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:20.405502 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:20.405531 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:20.408215 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:20.412244 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:20.412253 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:20.412279 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:20.412836 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:20.413263 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:20.413268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:20.413272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:20.413480 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:20.414195 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:20.414203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:20.414560 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 15:01:20.414891 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 15:01:20.415396 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22734,"thread_start_us":22525,"threads_started":1}
I20260925 15:01:20.417941 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:20.373225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:35647 (local address 127.4.243.254:35227)
0925 15:01:20.395099 (+ 21874us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:20.395109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:20.396196 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 15:01:20.397726 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:20.397732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:20.397818 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:20.397940 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:20.399563 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:20.400697 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:20.403268 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:20.403938 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:20.404071 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:20.408479 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:20.408485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:20.408490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:20.408545 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:20.412078 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:20.412998 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:20.413003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:20.413007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:20.413139 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:20.413603 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:20.413608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:20.413612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:20.413986 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:20.414130 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:20.417403 (+ 3273us) server_negotiation.cc:299] Negotiation successful
0925 15:01:20.417613 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21691,"thread_start_us":21557,"threads_started":1}
I20260925 15:01:20.419238 18274 heartbeater.cc:347] Connected to a master server at 127.4.243.254:35227
I20260925 15:01:20.419600 18274 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:20.420393 18274 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
I20260925 15:01:20.424329 17933 ts_manager.cc:194] Registered new tserver with Master: d771f639e51f499fab0225e07ca24ac7 (127.4.243.194:36679)
I20260925 15:01:20.426458 17933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:35647
I20260925 15:01:20.738549 18130 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:20.738987 18130 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
I20260925 15:01:20.740106 17933 ts_manager.cc:194] Registered new tserver with Master: 4c1fd57ac31342a8b0b13051a0b7499c (127.4.243.193:44945)
I20260925 15:01:20.741290 17933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:41837
Sep 25 15:01:20 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
W20260925 15:01:20.864995 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:20.865404 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:20.865576 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:20.877075 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:20.877214 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:20.877401 18276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:01:20.893165 18276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.243.212:58713/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:42071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=34731
--tserver_master_addrs=127.4.243.254:35227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:20.894989 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:20.896684 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:20.927697 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:20.927615 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:20.929733 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:20.930083 18276 server_base.cc:1034] running on GCE node
I20260925 15:01:20.930990 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:20.934473 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:20.935688 18276 hybrid_clock.cc:648] HybridClock initialized: now 1790348480935620 us; error 64 us; skew 500 ppm
I20260925 15:01:20.936030 18276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:20 dist-test-slave-qxh0 krb5kdc[16922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:20.939366 18276 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:01:20.942968 18276 webserver.cc:460] Webserver started at http://127.4.243.195:34731/ using document root <none> and password file <none>
I20260925 15:01:20.943913 18276 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:20.944090 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:01:20 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
I20260925 15:01:20.988099 18276 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.004s sys 0.006s
I20260925 15:01:20.991050 18295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:20.992544 18276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:01:20.992714 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "ba018b4110634fb58384a5b49530669c"
format_stamp: "Formatted at 2026-09-25 15:01:17 on dist-test-slave-qxh0"
server_key: "a95ef4df1183ce801f6e9fef3829d780"
server_key_iv: "d5c026a9c6219d12f76ddac76a68e887"
server_key_version: "kuduclusterkey@0"
I20260925 15:01:20.993675 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:21.026044 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:21.030526 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:21.030828 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:21.031908 18276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:21.033356 18276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:21.033464 18276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:21.033595 18276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:21.033687 18276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:21.094653 18276 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42071
I20260925 15:01:21.094669 18408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42071 every 8 connection(s)
I20260925 15:01:21.096138 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:21.101586 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18276
Sep 25 15:01:21 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for HTTP/127.4.243.212@KRBTEST.COM
Sep 25 15:01:21 dist-test-slave-qxh0 krb5kdc[16922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348480, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:21.162091 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.101197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:51157 (local address 127.4.243.254:35227)
0925 15:01:21.130706 (+ 29509us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.130713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.137884 (+ 7171us) server_negotiation.cc:407] Connection header received
0925 15:01:21.140730 (+ 2846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.140735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.140812 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.140908 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:21.143181 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.144368 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.147878 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.148528 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.148649 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.153229 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:21.153237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:21.153245 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:21.153290 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:21.158048 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.159054 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.159061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.159066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.159215 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.159777 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.159782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.159785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.160175 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:21.160330 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.161379 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.161611 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29299,"thread_start_us":29637,"threads_started":1}
I20260925 15:01:21.164234 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.103033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.4.243.254:35227)
0925 15:01:21.131073 (+ 28040us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.131079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.131098 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:01:21.131168 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.131173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.131264 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.131372 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:21.133462 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.139266 (+ 5804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.144510 (+ 5244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.145086 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.145212 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.157235 (+ 12023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:21.157241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:21.157246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:21.157441 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:21.160832 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.161793 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.161798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.161802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.161936 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.162406 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.162412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.162415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.162875 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:21.163101 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.163684 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.163898 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27929,"spinlock_wait_cycles":274688,"thread_start_us":32989,"threads_started":1}
I20260925 15:01:21.164955 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.100475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35227 (local address 127.4.243.195:51157)
0925 15:01:21.134397 (+ 33922us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.134454 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:21.140282 (+ 5828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:21.141364 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:21.141377 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:21.142000 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:21.142910 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.142935 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.145901 (+ 2966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.145906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.147709 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.147723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.149010 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.149019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.149323 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.150195 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:21.150235 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:21.152956 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:21.158209 (+ 5253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:21.158220 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:21.158240 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:21.158876 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:21.159375 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:21.159381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:21.159387 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:21.159634 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:21.160424 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:21.160435 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:21.160889 (+ 454us) client_negotiation.cc:769] Sending connection context
0925 15:01:21.162629 (+ 1740us) client_negotiation.cc:240] Negotiation successful
0925 15:01:21.163248 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33586,"thread_start_us":33345,"threads_started":1}
I20260925 15:01:21.167122 18409 heartbeater.cc:347] Connected to a master server at 127.4.243.254:35227
I20260925 15:01:21.167455 18409 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:21.168387 18409 heartbeater.cc:511] Master 127.4.243.254:35227 requested a full tablet report, sending...
I20260925 15:01:21.170786 17932 ts_manager.cc:194] Registered new tserver with Master: ba018b4110634fb58384a5b49530669c (127.4.243.195:42071)
I20260925 15:01:21.171849 17932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:51157
I20260925 15:01:21.183104 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:21.209103 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.188580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.4.243.254:35227)
0925 15:01:21.188771 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.188777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.191319 (+ 2542us) server_negotiation.cc:407] Connection header received
0925 15:01:21.191528 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.191535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.191616 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.191693 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:21.193910 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.195063 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.197978 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.198649 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.198765 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.202169 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:21.202176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:21.202181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:21.202222 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:21.205849 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.206821 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.206826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.206830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.206975 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.207406 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.207411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.207414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.207739 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:21.207957 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.208501 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.208779 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:01:21.215740 17932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:21.226969 17932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:21.275420 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.264230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59552)
0925 15:01:21.265305 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.265322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:21.265439 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:21.266957 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:21.266965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:21.267006 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:21.267407 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.267420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.270038 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.270043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.272436 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.272451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.274517 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.274524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.274733 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.274789 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:01:21.274902 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:01:21.275086 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":841,"threads_started":1}
I20260925 15:01:21.276347 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.262871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.4.243.193:44945)
0925 15:01:21.264226 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.264235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.265549 (+ 1314us) server_negotiation.cc:407] Connection header received
0925 15:01:21.265647 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.265655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.265940 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.266133 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:21.267636 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.269282 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.272613 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.274024 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.274149 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.274378 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.275824 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.276003 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":1029,"threads_started":1}
I20260925 15:01:21.280462 18065 tablet_service.cc:1467] Processing CreateTablet for tablet ea09ff1e3c95411299d42b4d4924fa2f (DEFAULT_TABLE table=test-table [id=63baf0bc4f4646caa819fbda6a085ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:21.282065 18065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea09ff1e3c95411299d42b4d4924fa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:21.284034 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.246333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42071 (local address 127.0.0.1:55534)
0925 15:01:21.262009 (+ 15676us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.262063 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:21.262266 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:21.273599 (+ 11333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:21.273603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:21.273627 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:21.273956 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.273968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.278625 (+ 4657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.278630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.280896 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.280910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.283046 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.283053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.283230 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.283264 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:01:21.283336 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:01:21.283473 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15422,"mutex_wait_us":49,"thread_start_us":15260,"threads_started":1}
I20260925 15:01:21.285331 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.246434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:36679 (local address 127.0.0.1:60898)
0925 15:01:21.262818 (+ 16384us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.262838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:21.262942 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:21.271790 (+ 8848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:21.271795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:21.271821 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:21.272185 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.272196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.278540 (+ 6344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.278545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.280998 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.281014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.282914 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.282921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.283771 (+ 850us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.283804 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:01:21.283920 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:01:21.284144 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16270,"mutex_wait_us":59,"thread_start_us":16082,"threads_started":1}
I20260925 15:01:21.285365 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.246938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.4.243.194:36679)
0925 15:01:21.270034 (+ 23096us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.270043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.270073 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:01:21.270168 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.270176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.270425 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.272382 (+ 1957us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:21.275815 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.278396 (+ 2581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.281186 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.282664 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.284676 (+ 2012us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.284866 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.284930 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.285089 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22939,"thread_start_us":22846,"threads_started":1}
I20260925 15:01:21.286657 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.246959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.4.243.195:42071)
0925 15:01:21.271060 (+ 24101us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.271069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.271104 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:01:21.271190 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.271198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.271391 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.274152 (+ 2761us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:21.274445 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.278099 (+ 3654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.281089 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.282779 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.283665 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.286096 (+ 2431us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.286167 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.286311 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23907,"thread_start_us":24572,"threads_started":1}
I20260925 15:01:21.289193 18209 tablet_service.cc:1467] Processing CreateTablet for tablet ea09ff1e3c95411299d42b4d4924fa2f (DEFAULT_TABLE table=test-table [id=63baf0bc4f4646caa819fbda6a085ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:21.290509 18209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea09ff1e3c95411299d42b4d4924fa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:21.293047 18343 tablet_service.cc:1467] Processing CreateTablet for tablet ea09ff1e3c95411299d42b4d4924fa2f (DEFAULT_TABLE table=test-table [id=63baf0bc4f4646caa819fbda6a085ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:21.294377 18343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea09ff1e3c95411299d42b4d4924fa2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:21.307008 18430 tablet_bootstrap.cc:492] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: Bootstrap starting.
I20260925 15:01:21.310472 18430 tablet_bootstrap.cc:654] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:21.311746 18430 log.cc:824] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:21.312624 18431 tablet_bootstrap.cc:492] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Bootstrap starting.
I20260925 15:01:21.314854 18432 tablet_bootstrap.cc:492] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Bootstrap starting.
I20260925 15:01:21.321866 18432 tablet_bootstrap.cc:654] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:21.323285 18431 tablet_bootstrap.cc:654] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:21.323244 18432 log.cc:824] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:21.331701 18431 log.cc:824] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:21.336565 18431 tablet_bootstrap.cc:492] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: No bootstrap required, opened a new log
I20260925 15:01:21.336887 18431 ts_tablet_manager.cc:1397] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.003s
I20260925 15:01:21.337880 18430 tablet_bootstrap.cc:492] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: No bootstrap required, opened a new log
I20260925 15:01:21.338196 18430 ts_tablet_manager.cc:1397] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.000s
I20260925 15:01:21.349790 18430 raft_consensus.cc:348] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.350396 18430 raft_consensus.cc:374] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:21.350549 18430 raft_consensus.cc:729] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1fd57ac31342a8b0b13051a0b7499c, State: Initialized, Role: FOLLOWER
I20260925 15:01:21.351380 18430 consensus_queue.cc:260] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.359443 18430 ts_tablet_manager.cc:1428] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: Time spent starting tablet: real 0.021s user 0.018s sys 0.000s
I20260925 15:01:21.359823 18130 heartbeater.cc:503] Master 127.4.243.254:35227 was elected leader, sending a full tablet report...
I20260925 15:01:21.363466 18432 tablet_bootstrap.cc:492] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: No bootstrap required, opened a new log
I20260925 15:01:21.367268 18432 ts_tablet_manager.cc:1397] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Time spent bootstrapping tablet: real 0.053s user 0.017s sys 0.026s
I20260925 15:01:21.376959 18432 raft_consensus.cc:348] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.377449 18432 raft_consensus.cc:374] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:21.377573 18432 raft_consensus.cc:729] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba018b4110634fb58384a5b49530669c, State: Initialized, Role: FOLLOWER
I20260925 15:01:21.378301 18432 consensus_queue.cc:260] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.379400 18409 heartbeater.cc:503] Master 127.4.243.254:35227 was elected leader, sending a full tablet report...
I20260925 15:01:21.380084 18432 ts_tablet_manager.cc:1428] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260925 15:01:21.382583 18431 raft_consensus.cc:348] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.383406 18431 raft_consensus.cc:374] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:21.383529 18431 raft_consensus.cc:729] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d771f639e51f499fab0225e07ca24ac7, State: Initialized, Role: FOLLOWER
I20260925 15:01:21.384176 18431 consensus_queue.cc:260] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.385293 18274 heartbeater.cc:503] Master 127.4.243.254:35227 was elected leader, sending a full tablet report...
I20260925 15:01:21.387029 18431 ts_tablet_manager.cc:1428] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Time spent starting tablet: real 0.050s user 0.013s sys 0.000s
I20260925 15:01:21.417778 18436 raft_consensus.cc:482] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:21.418128 18436 raft_consensus.cc:504] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.420336 18436 leader_election.cc:288] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d771f639e51f499fab0225e07ca24ac7 (127.4.243.194:36679), ba018b4110634fb58384a5b49530669c (127.4.243.195:42071)
I20260925 15:01:21.428540 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.421369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:56175 (local address 127.4.243.194:36679)
0925 15:01:21.421830 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.421840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.422679 (+ 839us) server_negotiation.cc:407] Connection header received
0925 15:01:21.423032 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.423038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.423117 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.423225 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:21.423960 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.424881 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.426525 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.427545 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.427713 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.427792 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.428139 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.428259 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260925 15:01:21.428663 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.420813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:36679 (local address 127.4.243.193:56175)
0925 15:01:21.422500 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.422532 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:21.422823 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:21.423272 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:21.423280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:21.423327 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:21.423796 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.423806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.425019 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.425024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.426364 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.426376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.427759 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.427765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.427980 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.428005 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:01:21.428115 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:01:21.428237 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"mutex_wait_us":29,"thread_start_us":2716,"threads_started":1}
I20260925 15:01:21.429741 18229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea09ff1e3c95411299d42b4d4924fa2f" candidate_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d771f639e51f499fab0225e07ca24ac7" is_pre_election: true
I20260925 15:01:21.430341 18229 raft_consensus.cc:2454] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1fd57ac31342a8b0b13051a0b7499c in term 0.
I20260925 15:01:21.430938 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.422201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:59865 (local address 127.4.243.195:42071)
0925 15:01:21.422402 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.422408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.423977 (+ 1569us) server_negotiation.cc:407] Connection header received
0925 15:01:21.424265 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.424271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.424345 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.424436 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:21.425047 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.426108 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.428627 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.429792 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.429935 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.430005 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.430519 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.430690 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:01:21.431036 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.421593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42071 (local address 127.4.243.193:59865)
0925 15:01:21.423836 (+ 2243us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.423855 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:21.424099 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:21.424482 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:21.424487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:21.424515 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:21.424851 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.424863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.426290 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.426295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.428446 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:21.428460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.430191 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.430198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:21.430356 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.430387 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:01:21.430491 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:01:21.430634 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2108,"mutex_wait_us":65,"thread_start_us":811,"threads_started":1}
I20260925 15:01:21.431131 18019 leader_election.cc:302] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1fd57ac31342a8b0b13051a0b7499c, d771f639e51f499fab0225e07ca24ac7; no voters:
I20260925 15:01:21.431556 18436 raft_consensus.cc:2781] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:21.431699 18436 raft_consensus.cc:482] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:21.431793 18436 raft_consensus.cc:3037] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:21.432041 18363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea09ff1e3c95411299d42b4d4924fa2f" candidate_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba018b4110634fb58384a5b49530669c" is_pre_election: true
I20260925 15:01:21.432492 18363 raft_consensus.cc:2454] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1fd57ac31342a8b0b13051a0b7499c in term 0.
I20260925 15:01:21.434821 18436 raft_consensus.cc:504] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.435467 18436 leader_election.cc:288] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [CANDIDATE]: Term 1 election: Requested vote from peers d771f639e51f499fab0225e07ca24ac7 (127.4.243.194:36679), ba018b4110634fb58384a5b49530669c (127.4.243.195:42071)
I20260925 15:01:21.435838 18229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea09ff1e3c95411299d42b4d4924fa2f" candidate_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d771f639e51f499fab0225e07ca24ac7"
I20260925 15:01:21.435895 18363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea09ff1e3c95411299d42b4d4924fa2f" candidate_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba018b4110634fb58384a5b49530669c"
I20260925 15:01:21.436043 18229 raft_consensus.cc:3037] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:21.436074 18363 raft_consensus.cc:3037] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:21.438158 18363 raft_consensus.cc:2454] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1fd57ac31342a8b0b13051a0b7499c in term 1.
I20260925 15:01:21.438552 18020 leader_election.cc:302] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1fd57ac31342a8b0b13051a0b7499c, ba018b4110634fb58384a5b49530669c; no voters:
I20260925 15:01:21.438854 18436 raft_consensus.cc:2781] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:21.439029 18229 raft_consensus.cc:2454] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1fd57ac31342a8b0b13051a0b7499c in term 1.
I20260925 15:01:21.439442 18436 raft_consensus.cc:686] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 1 LEADER]: Becoming Leader. State: Replica: 4c1fd57ac31342a8b0b13051a0b7499c, State: Running, Role: LEADER
I20260925 15:01:21.440204 18436 consensus_queue.cc:237] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } }
I20260925 15:01:21.445209 17932 catalog_manager.cc:5495] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1fd57ac31342a8b0b13051a0b7499c (127.4.243.193). New cstate: current_term: 1 leader_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1fd57ac31342a8b0b13051a0b7499c" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44945 } health_report { overall_health: HEALTHY } } }
W20260925 15:01:21.451414 18131 tablet.cc:2369] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:21.471925 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.466389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.4.243.193:44945)
0925 15:01:21.466650 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.466657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.466738 (+ 81us) server_negotiation.cc:407] Connection header received
0925 15:01:21.467007 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.467011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.467097 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.467253 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:21.467638 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.468379 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.469762 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.470168 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.470301 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.470830 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:21.471335 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:21.471499 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.471577 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.471666 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:01:21.480301 18229 raft_consensus.cc:1261] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 1 FOLLOWER]: Refusing update from remote peer 4c1fd57ac31342a8b0b13051a0b7499c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:21.480414 18363 raft_consensus.cc:1261] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 1 FOLLOWER]: Refusing update from remote peer 4c1fd57ac31342a8b0b13051a0b7499c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:21.481402 18448 consensus_queue.cc:1035] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d771f639e51f499fab0225e07ca24ac7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:21.482164 18436 consensus_queue.cc:1035] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba018b4110634fb58384a5b49530669c" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:21.498327 18453 mvcc.cc:205] Tried to move back new op lower bound from 7333267380132913152 to 7333267379982540800. Current Snapshot: MvccSnapshot[applied={T|T < 7333267380132913152}]
I20260925 15:01:21.502547 18452 mvcc.cc:205] Tried to move back new op lower bound from 7333267380132913152 to 7333267379982540800. Current Snapshot: MvccSnapshot[applied={T|T < 7333267380132913152}]
I20260925 15:01:21.516358 17933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:21.516624 17933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:21.520569 17933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f259776446434c6c94e15019ea94848b: Sending DeleteTablet for 3 replicas of tablet ea09ff1e3c95411299d42b4d4924fa2f
I20260925 15:01:21.521744 18209 tablet_service.cc:1514] Processing DeleteTablet for tablet ea09ff1e3c95411299d42b4d4924fa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:21 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:60898
I20260925 15:01:21.522254 18065 tablet_service.cc:1514] Processing DeleteTablet for tablet ea09ff1e3c95411299d42b4d4924fa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:21 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:59552
I20260925 15:01:21.523530 18343 tablet_service.cc:1514] Processing DeleteTablet for tablet ea09ff1e3c95411299d42b4d4924fa2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:21 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:55534
I20260925 15:01:21.532743 18463 tablet_replica.cc:330] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: stopping tablet replica
I20260925 15:01:21.533223 18463 raft_consensus.cc:2229] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:21.533468 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 17970
I20260925 15:01:21.533785 18463 raft_consensus.cc:2258] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:21.536092 18463 ts_tablet_manager.cc:1905] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:21.538981 18462 tablet_replica.cc:330] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: stopping tablet replica
I20260925 15:01:21.539350 18462 raft_consensus.cc:2229] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:21.540010 18462 raft_consensus.cc:2258] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:21.543171 18463 ts_tablet_manager.cc:1918] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:21.543313 18463 log.cc:1196] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/ea09ff1e3c95411299d42b4d4924fa2f
I20260925 15:01:21.543879 18463 ts_tablet_manager.cc:1939] T ea09ff1e3c95411299d42b4d4924fa2f P ba018b4110634fb58384a5b49530669c: Deleting consensus metadata
I20260925 15:01:21.545698 17920 catalog_manager.cc:4841] TS ba018b4110634fb58384a5b49530669c (127.4.243.195:42071): tablet ea09ff1e3c95411299d42b4d4924fa2f (table test-table [id=63baf0bc4f4646caa819fbda6a085ffb]) successfully deleted
I20260925 15:01:21.547386 18462 ts_tablet_manager.cc:1905] T ea09ff1e3c95411299d42b4d4924fa2f P 4c1fd57ac31342a8b0b13051a0b7499c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:21.550525 18461 tablet_replica.cc:330] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: stopping tablet replica
I20260925 15:01:21.550940 18461 raft_consensus.cc:2229] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:21.551452 18461 raft_consensus.cc:2258] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:21.553521 18461 ts_tablet_manager.cc:1905] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:21.556173 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.535187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.4.243.193:44945)
0925 15:01:21.535353 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.535359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.535375 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:01:21.535457 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.535462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.535542 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.535640 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:21.537893 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.539156 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.542440 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.543109 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.543230 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.547555 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:21.547565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:21.547582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:21.547621 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:21.552210 (+ 4589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.553142 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.553151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.553157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.553303 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.553789 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.553793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.553797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.554217 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:21.554508 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.555609 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.555834 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:01:21.557149 18125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:21.561033 18461 ts_tablet_manager.cc:1918] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:21.561220 18461 log.cc:1196] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/ea09ff1e3c95411299d42b4d4924fa2f
I20260925 15:01:21.561846 18461 ts_tablet_manager.cc:1939] T ea09ff1e3c95411299d42b4d4924fa2f P d771f639e51f499fab0225e07ca24ac7: Deleting consensus metadata
I20260925 15:01:21.564131 17919 catalog_manager.cc:4841] TS d771f639e51f499fab0225e07ca24ac7 (127.4.243.194:36679): tablet ea09ff1e3c95411299d42b4d4924fa2f (table test-table [id=63baf0bc4f4646caa819fbda6a085ffb]) successfully deleted
I20260925 15:01:21.659035 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 17970
W20260925 15:01:21.676565 17920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:44945 (error 108)
I20260925 15:01:21.676713 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18134
I20260925 15:01:21.678738 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.678172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59574)
0925 15:01:21.678359 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.678471 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 15:01:21.679029 17920 catalog_manager.cc:4584] TS 4c1fd57ac31342a8b0b13051a0b7499c (127.4.243.193:44945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea09ff1e3c95411299d42b4d4924fa2f: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
I20260925 15:01:21.691234 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.677487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.4.243.194:36679)
0925 15:01:21.677662 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.677669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.678301 (+ 632us) server_negotiation.cc:407] Connection header received
0925 15:01:21.678360 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.678363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.678420 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.678543 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:21.680165 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.680868 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.682151 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.682561 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.682768 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.685303 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:21.685327 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:21.685343 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:21.685377 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:21.688417 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.689138 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.689148 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.689151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.689286 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.689647 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.689653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.689656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.690036 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:21.690324 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.690655 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.690890 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:01:21.692128 18269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:21.732780 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.732210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59576)
0925 15:01:21.732473 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.732587 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 15:01:21.733887 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.733522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59592)
0925 15:01:21.733674 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.733765 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:01:21.780159 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18134
I20260925 15:01:21.794319 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.793759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59594)
0925 15:01:21.794010 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.794123 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:01:21.795374 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.794951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59610)
0925 15:01:21.795128 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.795225 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:01:21.796141 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18276
I20260925 15:01:21.809376 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.796734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.4.243.195:42071)
0925 15:01:21.796905 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:21.796913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:21.796934 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:01:21.797166 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:21.797170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:21.797239 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:21.797338 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:21.798556 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.799308 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.800539 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:21.800915 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:21.801049 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:21.803763 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:21.803779 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:21.803796 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:21.803831 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:21.806687 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.807373 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.807394 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.807396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.807531 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:21.807903 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:21.807909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:21.807912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:21.808250 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:21.808579 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:21.808838 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:01:21.809055 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:01:21.810084 18403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:21.876214 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.875667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59616)
0925 15:01:21.875949 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.876041 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 15:01:21.877197 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:21.876844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44945 (local address 127.0.0.1:59628)
0925 15:01:21.877018 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:21.877096 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:01:21.897590 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18276
I20260925 15:01:21.914363 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 17900
I20260925 15:01:21.915521 17963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:22.036665 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 17900
I20260925 15:01:22.060781 5071 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:01:22.395241 5071 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:01:22.395466 5071 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:01:22.740020 5071 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:01:22.740 UTC [16952] LOG: received smart shutdown request
2026-09-25 15:01:22.743 UTC [16952] LOG: background worker "logical replication launcher" (PID 16961) exited with exit code 1
2026-09-25 15:01:22.743 UTC [16956] LOG: shutting down
2026-09-25 15:01:22.909 UTC [16952] LOG: database system is shut down
2026-09-25T15:01:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51535 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:01:23 dist-test-slave-qxh0 krb5kdc[18477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:01:23 dist-test-slave-qxh0 krb5kdc[18477](info): set up 2 sockets
Sep 25 15:01:23 dist-test-slave-qxh0 krb5kdc[18477](info): commencing operation
krb5kdc: starting...
W20260925 15:01:24.114811 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:01:24 dist-test-slave-qxh0 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:01:24.329983 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35207
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35207,127.4.243.253:32777,127.4.243.252:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:24.649263 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:24.649654 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:24.649827 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:24.656766 18490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:24.656913 18490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:24.657022 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:24.657122 18490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:24.657207 18490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:24.666495 18490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35207,127.4.243.253:32777,127.4.243.252:40747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:24.667815 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:24.669119 18490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:24.680886 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:24.680845 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:24.681452 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:24.682348 18490 server_base.cc:1034] running on GCE node
I20260925 15:01:24.683228 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:24.684577 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:24.685819 18490 hybrid_clock.cc:648] HybridClock initialized: now 1790348484685731 us; error 67 us; skew 500 ppm
I20260925 15:01:24.686190 18490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:24 dist-test-slave-qxh0 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:24.689363 18490 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:24.690886 18490 webserver.cc:460] Webserver started at http://127.4.243.254:44769/ using document root <none> and password file <none>
I20260925 15:01:24.691560 18490 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:24.691701 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:24.692008 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:24.694955 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "d82d937710954a23b812a8948d7c2957"
format_stamp: "Formatted at 2026-09-25 15:01:24 on dist-test-slave-qxh0"
server_key: "f477c0330b401ec0297262469eb6076c"
server_key_iv: "ce069dcfcdd65777fb73d429e88a15b7"
server_key_version: "encryptionkey@0"
I20260925 15:01:24.695741 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "d82d937710954a23b812a8948d7c2957"
format_stamp: "Formatted at 2026-09-25 15:01:24 on dist-test-slave-qxh0"
server_key: "f477c0330b401ec0297262469eb6076c"
server_key_iv: "ce069dcfcdd65777fb73d429e88a15b7"
server_key_version: "encryptionkey@0"
I20260925 15:01:24.700551 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:01:24.703727 18505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:24.705005 18490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:24.705184 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "d82d937710954a23b812a8948d7c2957"
format_stamp: "Formatted at 2026-09-25 15:01:24 on dist-test-slave-qxh0"
server_key: "f477c0330b401ec0297262469eb6076c"
server_key_iv: "ce069dcfcdd65777fb73d429e88a15b7"
server_key_version: "encryptionkey@0"
I20260925 15:01:24.705365 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:24.719784 18490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:24.723584 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:24.723863 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:24.741896 18490 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:35207
I20260925 15:01:24.741974 18556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:35207 every 8 connection(s)
I20260925 15:01:24.743225 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:24.746587 18557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:24.750538 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18490
I20260925 15:01:24.750793 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:01:24.751462 5071 external_mini_cluster.cc:1428] Setting key de5dea19216a34ea0358486cb49c2d46
I20260925 15:01:24.752877 18557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:01:24 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.253@KRBTEST.COM" created.
I20260925 15:01:24.774904 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.755858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35207 (local address 127.0.0.1:57614)
0925 15:01:24.756591 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.756633 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:24.758577 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:24.759400 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:24.759406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:24.760053 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:24.760376 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:24.760392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:24.761965 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:24.761972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:24.763100 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:24.763112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:24.764074 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:24.764081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:24.764312 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:24.765009 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:24.765030 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:24.767476 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:24.770629 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:24.770639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:24.770650 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:24.771082 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:24.771492 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:24.771496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:24.771500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:24.771650 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:24.772446 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:24.772453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:24.772704 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 15:01:24.772956 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 15:01:24.773539 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":167,"thread_start_us":309,"threads_started":1}
I20260925 15:01:24.774950 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.757081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.4.243.254:35207)
0925 15:01:24.757787 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:24.757794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:24.757875 (+ 81us) server_negotiation.cc:407] Connection header received
0925 15:01:24.758970 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:24.758975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:24.759175 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:24.759373 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:24.760579 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:24.761805 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:24.763236 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:24.763770 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:24.764022 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:24.767694 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:24.767701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:24.767713 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:24.767739 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:24.770471 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:24.771215 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:24.771221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:24.771223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:24.771374 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:24.771818 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:24.771829 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:24.771831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:24.772301 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:24.772510 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:24.773086 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 15:01:24.773438 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":518,"thread_start_us":351,"threads_started":1}
I20260925 15:01:24.778141 18557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:01:24.779240 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.778875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40148)
0925 15:01:24.779035 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.779135 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 15:01:24.779527 18506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
I20260925 15:01:24.781126 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.780719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40164)
0925 15:01:24.780872 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.780947 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 15:01:24.781314 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32777: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab.
Entry for principal kudu/127.4.243.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab.
I20260925 15:01:24.799973 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:32777
--webserver_interface=127.4.243.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab
--principal=kudu/127.4.243.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35207,127.4.243.253:32777,127.4.243.252:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:24.830673 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } attempt: 1
I20260925 15:01:24.831919 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.831387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40180)
0925 15:01:24.831603 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.831749 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:01:24.832918 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.832544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40190)
0925 15:01:24.832686 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.832764 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:01:24.833112 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32777: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
I20260925 15:01:24.901401 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } attempt: 2
I20260925 15:01:24.902707 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.902097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40200)
0925 15:01:24.902377 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.902492 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 15:01:24.903756 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.903376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40204)
0925 15:01:24.903531 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.903613 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:01:24.903956 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32777: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
I20260925 15:01:24.995227 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } attempt: 3
I20260925 15:01:24.996609 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.995995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40212)
0925 15:01:24.996269 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.996420 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 15:01:24.997680 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:24.997322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40218)
0925 15:01:24.997465 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:24.997539 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:01:24.997901 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32777: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
W20260925 15:01:25.113905 18564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:25.114144 18564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:25.114286 18564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:25.120993 18564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:25.121091 18564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:25.121191 18564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:25.121287 18564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:25.121377 18564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:25.130829 18564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35207,127.4.243.253:32777,127.4.243.252:40747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab
--principal=kudu/127.4.243.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.243.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:25.132020 18564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:25.133224 18564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:01:25.141161 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } attempt: 4
I20260925 15:01:25.142424 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.141776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40234)
0925 15:01:25.142133 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.142243 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260925 15:01:25.143699 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.143213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40250)
0925 15:01:25.143443 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.143551 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260925 15:01:25.143934 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32777: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32777: connect: Connection refused (error 111)
W20260925 15:01:25.143949 18569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:25.144208 18570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:25.144613 18572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:25.145347 18564 server_base.cc:1034] running on GCE node
I20260925 15:01:25.146102 18564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:25.147399 18564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:25.148586 18564 hybrid_clock.cc:648] HybridClock initialized: now 1790348485148545 us; error 27 us; skew 500 ppm
I20260925 15:01:25.148921 18564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:25.152014 18564 init.cc:377] Logged in from keytab as kudu/127.4.243.253@KRBTEST.COM (short username kudu)
I20260925 15:01:25.153288 18564 webserver.cc:460] Webserver started at http://127.4.243.253:33617/ using document root <none> and password file <none>
I20260925 15:01:25.153877 18564 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:25.153966 18564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:25.154184 18564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:25.156844 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data/instance:
uuid: "36349d8837d54b14b01f5b1ad49e6e66"
format_stamp: "Formatted at 2026-09-25 15:01:25 on dist-test-slave-qxh0"
server_key: "551cd0d4b6ac21c6aea0ccae7028165a"
server_key_iv: "8e407da69a03f4b3a1bde245963f6956"
server_key_version: "encryptionkey@0"
I20260925 15:01:25.157545 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/wal/instance:
uuid: "36349d8837d54b14b01f5b1ad49e6e66"
format_stamp: "Formatted at 2026-09-25 15:01:25 on dist-test-slave-qxh0"
server_key: "551cd0d4b6ac21c6aea0ccae7028165a"
server_key_iv: "8e407da69a03f4b3a1bde245963f6956"
server_key_version: "encryptionkey@0"
I20260925 15:01:25.162197 18564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:01:25.165947 18579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:25.167296 18564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:25.167477 18564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/wal
uuid: "36349d8837d54b14b01f5b1ad49e6e66"
format_stamp: "Formatted at 2026-09-25 15:01:25 on dist-test-slave-qxh0"
server_key: "551cd0d4b6ac21c6aea0ccae7028165a"
server_key_iv: "8e407da69a03f4b3a1bde245963f6956"
server_key_version: "encryptionkey@0"
I20260925 15:01:25.167665 18564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:25.192386 18564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:25.196739 18564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:25.197000 18564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:25.214942 18564 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.253:32777
I20260925 15:01:25.214983 18630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.253:32777 every 8 connection(s)
I20260925 15:01:25.216207 18564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
I20260925 15:01:25.219818 18631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:25.220582 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18564
I20260925 15:01:25.220758 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-1/wal/instance
I20260925 15:01:25.221233 5071 external_mini_cluster.cc:1428] Setting key 7f36fafe9c860bec848ae6845a023c70
I20260925 15:01:25.226164 18631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.252@KRBTEST.COM" created.
I20260925 15:01:25.245630 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.228667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.4.243.254:35207)
0925 15:01:25.228830 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.228834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.230132 (+ 1298us) server_negotiation.cc:407] Connection header received
0925 15:01:25.231942 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.231947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.232033 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.232148 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.233535 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.234188 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.235757 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.236156 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.236304 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.239934 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.239939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.239942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.239971 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.242673 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.243377 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.243384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.243387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.243508 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.243941 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.243946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.243949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.244234 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.244387 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.245078 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.245303 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:01:25.246587 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.229274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35207 (local address 127.0.0.1:57616)
0925 15:01:25.229967 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.230019 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.231674 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.232370 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.232377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.232858 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.233372 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.233387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.234348 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.234352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.235614 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.235625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.236397 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.236404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.236599 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.237273 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.237299 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.239739 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.242816 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.242824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.242839 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.243252 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.243618 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.243623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.243626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.243815 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.244379 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.244387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.244708 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.244972 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.245461 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":296,"threads_started":1}
I20260925 15:01:25.248270 18631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:01:25.261727 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.249043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:40266)
0925 15:01:25.249197 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.249216 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.249335 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.250055 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.250059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.250482 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.250635 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.250648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.251946 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.251954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.252763 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.252773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.253504 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.253509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.253740 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.254267 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.254300 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.256423 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.259449 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.259456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.259459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.259790 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.260125 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.260131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.260134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.260275 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.260813 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.260821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.260939 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.261076 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.261329 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":3}
I20260925 15:01:25.261973 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.249142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.4.243.253:32777)
0925 15:01:25.249695 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.249700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.249722 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:01:25.249759 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.249764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.249922 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.250067 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.250802 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.251814 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.252901 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.253362 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.253493 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.256602 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.256611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.256622 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.256649 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.259321 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.259911 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.259924 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.259926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.260026 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.260370 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.260376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.260381 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.260704 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.260904 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.261282 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.261496 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":425,"thread_start_us":289,"threads_started":1}
I20260925 15:01:25.264142 18631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:01:25.265125 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.264723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:49090)
0925 15:01:25.264870 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.264959 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:01:25.265405 18581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.266896 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.266573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:49096)
0925 15:01:25.266735 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.266806 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab.
Entry for principal kudu/127.4.243.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab.
W20260925 15:01:25.267094 18631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.268126 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:40747
--webserver_interface=127.4.243.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab
--principal=kudu/127.4.243.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35207,127.4.243.253:32777,127.4.243.252:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:25.316429 18631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 1
I20260925 15:01:25.317677 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.317134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:49104)
0925 15:01:25.317350 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.317495 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:01:25.318687 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.318265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:49108)
0925 15:01:25.318425 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.318511 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:01:25.318887 18631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.387113 18631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 2
I20260925 15:01:25.388370 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.387834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:49110)
0925 15:01:25.388054 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.388182 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:01:25.389427 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.389033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:49118)
0925 15:01:25.389184 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.389266 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:01:25.389659 18631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.443203 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } attempt: 5
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:01:25.458196 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.444181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.4.243.253:32777)
0925 15:01:25.444346 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.444350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.444370 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:01:25.444563 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.444566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.444650 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.444736 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.445697 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.446411 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.448154 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.448607 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.448715 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.452838 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.452867 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.452870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.452904 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.455536 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.456341 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.456346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.456348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.456474 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.456946 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.456950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.456952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.457237 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.457367 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.457794 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.457980 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:01:25.458320 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.444027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:47596)
0925 15:01:25.444254 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.444277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.444409 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.444799 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.444805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.445294 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.445540 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.445552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.446550 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.446555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.447972 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.447989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.448807 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.448816 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.449055 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.449763 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.449793 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.452638 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.455681 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.455685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.455689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.456198 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.456608 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.456615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.456618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.456797 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.457364 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.457369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.457549 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.457704 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.457972 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:01:25.459378 18557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:01:25.460427 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.459995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34182)
0925 15:01:25.460172 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.460263 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:01:25.461360 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.461006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34192)
0925 15:01:25.461150 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.461242 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:01:25.461561 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.480903 18631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 3
I20260925 15:01:25.482097 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.481568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34206)
0925 15:01:25.481785 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.481902 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:01:25.483345 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.482791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34220)
0925 15:01:25.483052 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.483163 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260925 15:01:25.483611 18631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.512792 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 1
I20260925 15:01:25.513890 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.513418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34232)
0925 15:01:25.513606 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.513717 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:01:25.514906 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.514547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34234)
0925 15:01:25.514708 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.514782 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 15:01:25.515115 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.583374 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 2
I20260925 15:01:25.584515 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.584050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34238)
0925 15:01:25.584260 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.584355 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:01:25.585558 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.585239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34250)
0925 15:01:25.585382 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.585466 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:01:25.585779 18557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
W20260925 15:01:25.590528 18638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:25.590875 18638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:25.591054 18638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:25.597919 18638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:25.598034 18638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:25.598114 18638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:25.598187 18638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:25.598268 18638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:25.607740 18638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35207,127.4.243.253:32777,127.4.243.252:40747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab
--principal=kudu/127.4.243.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.243.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:25.609112 18638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:25.610416 18638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:25.621721 18643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:25.621757 18644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:25.621784 18646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:25.622475 18638 server_base.cc:1034] running on GCE node
I20260925 15:01:25.623183 18638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:25.624363 18638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:25.625576 18638 hybrid_clock.cc:648] HybridClock initialized: now 1790348485625518 us; error 26 us; skew 500 ppm
I20260925 15:01:25.625965 18638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:01:25.626847 18631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 4
I20260925 15:01:25.627858 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.627437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34266)
0925 15:01:25.627635 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.627722 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:01:25.628811 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.628457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34274)
0925 15:01:25.628631 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.628710 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:01:25.629019 18631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:40747: Network error: Client connection negotiation failed: client connection to 127.4.243.252:40747: connect: Connection refused (error 111)
I20260925 15:01:25.629521 18638 init.cc:377] Logged in from keytab as kudu/127.4.243.252@KRBTEST.COM (short username kudu)
I20260925 15:01:25.630882 18638 webserver.cc:460] Webserver started at http://127.4.243.252:34499/ using document root <none> and password file <none>
I20260925 15:01:25.631522 18638 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:25.631664 18638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:25.631938 18638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:25.634678 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data/instance:
uuid: "c720f7abf7564ad184c3886e1aca2471"
format_stamp: "Formatted at 2026-09-25 15:01:25 on dist-test-slave-qxh0"
server_key: "dc5e8c3629552b12e37d2d17d86b4000"
server_key_iv: "d20fa42f6b1e3575aaafbee63eb58d7d"
server_key_version: "encryptionkey@0"
I20260925 15:01:25.635475 18638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/wal/instance:
uuid: "c720f7abf7564ad184c3886e1aca2471"
format_stamp: "Formatted at 2026-09-25 15:01:25 on dist-test-slave-qxh0"
server_key: "dc5e8c3629552b12e37d2d17d86b4000"
server_key_iv: "d20fa42f6b1e3575aaafbee63eb58d7d"
server_key_version: "encryptionkey@0"
I20260925 15:01:25.640012 18638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:01:25.643116 18653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:25.644304 18638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 15:01:25.644486 18638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/wal
uuid: "c720f7abf7564ad184c3886e1aca2471"
format_stamp: "Formatted at 2026-09-25 15:01:25 on dist-test-slave-qxh0"
server_key: "dc5e8c3629552b12e37d2d17d86b4000"
server_key_iv: "d20fa42f6b1e3575aaafbee63eb58d7d"
server_key_version: "encryptionkey@0"
I20260925 15:01:25.644667 18638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:25.664400 18638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:25.668393 18638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:25.668648 18638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:25.686805 18638 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.252:40747
I20260925 15:01:25.686854 18704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.252:40747 every 8 connection(s)
I20260925 15:01:25.688097 18638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
I20260925 15:01:25.691636 18705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:25.692013 18557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 3
I20260925 15:01:25.697021 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18638
I20260925 15:01:25.697227 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284481048-5071-0/minicluster-data/master-2/wal/instance
I20260925 15:01:25.697816 5071 external_mini_cluster.cc:1428] Setting key f674a61c037f0138c957073df2416a2a
I20260925 15:01:25.703059 18705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:01:25.716707 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.692623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34276)
0925 15:01:25.692836 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.692861 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.692970 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.696924 (+ 3954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.696929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.697420 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.697631 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.697642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.700267 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.700276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.702274 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.702293 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.704128 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.704137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.704380 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.705079 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.705109 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.708378 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.713728 (+ 5350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.713732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.713736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.714228 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.714765 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.714769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.714771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.714913 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.715907 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.715912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.716079 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.716259 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.716457 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:01:25.718899 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.693509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.4.243.252:40747)
0925 15:01:25.694227 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.694238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.694279 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:01:25.696204 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.696224 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.696600 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.696966 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.697804 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.700028 (+ 2224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.702528 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.703901 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.704823 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.708715 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.708725 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.708746 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.708789 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.713585 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.714356 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.714372 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.714381 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.714631 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.715020 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.715025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.715029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.715749 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.715925 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.717197 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.717602 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":325,"threads_started":1}
I20260925 15:01:25.724550 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.706653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35207 (local address 127.0.0.1:48566)
0925 15:01:25.708892 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.708929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.709124 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.709756 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.709761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.710415 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.710800 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.710815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.711786 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.711791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.713067 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.713078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.714005 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.714011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.714196 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.715098 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.715113 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.718646 (+ 3533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.721765 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.721776 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.721783 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.722246 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.722655 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.722661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.722665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.722802 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.723478 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.723485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.723647 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.723834 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.724143 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2018,"thread_start_us":1877,"threads_started":1}
I20260925 15:01:25.724647 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.707189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.4.243.254:35207)
0925 15:01:25.707738 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.707744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.709016 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 15:01:25.709296 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.709302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.709365 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.709484 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.710929 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.711634 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.713232 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.713629 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.713756 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.718834 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.718840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.718845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.718890 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.721632 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.722376 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.722382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.722385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.722512 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.722939 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.722943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.722945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.723332 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.723496 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.724007 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.724217 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404}
I20260925 15:01:25.726042 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.698712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.4.243.252:40747)
0925 15:01:25.708539 (+ 9827us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.708543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.708565 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:01:25.708609 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.708614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.708701 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.708795 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.711084 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.712088 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.714014 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.714675 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.714769 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.719428 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.719434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.719439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.719480 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.722952 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.723722 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.723727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.723730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.723916 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.724342 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.724347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.724352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.724672 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.724945 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.725433 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.725622 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9655,"thread_start_us":9490,"threads_started":1}
I20260925 15:01:25.726322 18705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:01:25.726868 18557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: Bootstrap starting.
I20260925 15:01:25.730521 18557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:25.731559 18557 log.cc:824] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:25.734138 18557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: No bootstrap required, opened a new log
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:25.742515 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.727131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32777 (local address 127.0.0.1:47612)
0925 15:01:25.727306 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.727323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.728029 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.728514 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.728519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.728980 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.729165 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.729178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.730542 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.730547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.731935 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.731951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.733054 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.733059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.733243 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.733806 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.733843 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.736257 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.739978 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.739983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.739987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.740467 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.740860 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.740865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.740869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.741031 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.741641 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.741646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.741794 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.741936 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.742190 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:01:25.742935 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.727511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.4.243.253:32777)
0925 15:01:25.727925 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.727932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.727948 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:01:25.728150 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.728155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.728224 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.728330 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.729332 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.730386 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.732102 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.732743 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.732835 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.736447 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.736453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.736458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.736494 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.739835 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.740589 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.740594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.740597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.740742 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.741142 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.741146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.741149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.741521 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.741676 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.742330 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.742553 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260925 15:01:25.743645 18705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:01:25.744598 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.727879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.4.243.254:35207)
0925 15:01:25.728047 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.728052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.728067 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:01:25.728111 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.728115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.728175 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.728274 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.729678 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.730433 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.732222 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.732892 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.732979 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.737438 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.737446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.737454 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.737500 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.741190 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.741952 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.741957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.741960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.742115 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.742508 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.742513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.742516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.742923 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.743166 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.744105 (+ 939us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.744300 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:01:25.755714 18557 raft_consensus.cc:348] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:25.756012 18557 raft_consensus.cc:374] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:25.756124 18557 raft_consensus.cc:729] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82d937710954a23b812a8948d7c2957, State: Initialized, Role: FOLLOWER
I20260925 15:01:25.756709 18557 consensus_queue.cc:260] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:25.757586 18712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:25.758025 18712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:01:25.758455 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.744253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34294)
0925 15:01:25.744370 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.744392 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.744475 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.744933 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.744936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.745243 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.745429 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.745439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.746718 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.746722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.748148 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.748162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.749124 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.749130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.749300 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.749837 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.749858 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.752274 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.755721 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.755726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.755731 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.756200 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.756595 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.756600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.756603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.756740 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.757447 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.757452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.757593 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.757760 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.758021 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:01:25.758936 18557 sys_catalog.cc:564] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:25.759349 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.744390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.4.243.252:40747)
0925 15:01:25.744532 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.744539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.744555 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:01:25.744598 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.744603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.744674 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.744761 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.745567 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.746614 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.748272 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.748952 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.749188 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.752451 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.752458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.752465 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.752502 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.755583 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.756328 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.756332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.756335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.756480 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.756846 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.756850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.756853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.757218 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.757366 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.758845 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.759037 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:01:25.763159 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.746205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.4.243.253:32777)
0925 15:01:25.746337 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.746342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.746487 (+ 145us) server_negotiation.cc:407] Connection header received
0925 15:01:25.746730 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.746734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.746800 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.746879 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.748507 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.749496 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.751260 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.751727 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.751857 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.756561 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.756569 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.756574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.756609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.760086 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.760959 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.760963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.760967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.761105 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.761620 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.761625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.761628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.761951 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.762207 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.762640 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.762845 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:01:25.764366 18705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471: Bootstrap starting.
I20260925 15:01:25.767614 18705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:25.768932 18705 log.cc:824] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:25.771850 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:01:25.772583 18705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471: No bootstrap required, opened a new log
W20260925 15:01:25.775792 18724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:01:25.775985 18724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:01:25.782351 18705 raft_consensus.cc:348] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:25.782869 18705 raft_consensus.cc:374] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:25.783001 18705 raft_consensus.cc:729] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c720f7abf7564ad184c3886e1aca2471, State: Initialized, Role: FOLLOWER
I20260925 15:01:25.783834 18705 consensus_queue.cc:260] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:25.776207 18490 master_runner.cc:186] Error getting master registration for 127.4.243.253:32777: OK, instance_id { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" instance_seqno: 1790348485209694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:01:25.786754 18490 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:01:25.798287 18727 sys_catalog.cc:455] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:25.798856 18727 sys_catalog.cc:458] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:01:25.799417 18705 sys_catalog.cc:564] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:25.820163 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.790731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.4.243.253:32777)
0925 15:01:25.791164 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.791171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.797676 (+ 6505us) server_negotiation.cc:407] Connection header received
0925 15:01:25.797895 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.797899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.797985 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.798052 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.800626 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.802118 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.804939 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.805915 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.806054 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.811011 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.811021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.811029 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.811086 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.815701 (+ 4615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.816895 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.816903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.816909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.817134 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.817698 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.817707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.817712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.818236 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.818550 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.819335 (+ 785us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.819649 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260925 15:01:25.833149 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.790999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.4.243.254:35207)
0925 15:01:25.791145 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.791151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.797407 (+ 6256us) server_negotiation.cc:407] Connection header received
0925 15:01:25.797645 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.797653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.797736 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.797829 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.809502 (+ 11673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.810955 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.813659 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.814606 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.814731 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.825072 (+ 10341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.825078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.825083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.825122 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.828754 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.829901 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.829908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.829932 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.830401 (+ 469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.830872 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.830890 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.830895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.831294 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.832358 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.832529 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.832788 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:01:25.835165 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.797831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.4.243.252:40747)
0925 15:01:25.798008 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.798013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.817630 (+ 19617us) server_negotiation.cc:407] Connection header received
0925 15:01:25.817787 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.817795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.817881 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.817989 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.819749 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.820855 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.823967 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.824669 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.824763 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.828200 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.828206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.828212 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.828257 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.831791 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.832722 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.832727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.832731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.832863 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.833269 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.833276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.833279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.833630 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.833849 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.834334 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.834554 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 15:01:25.845242 18744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:01:25.845407 18744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:01:25.843164 18638 master_runner.cc:186] Error getting master registration for 127.4.243.253:32777: OK, instance_id { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" instance_seqno: 1790348485209694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:01:25.854521 18638 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:01:25.920697 18659 tablet.cc:2369] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:25.928237 18631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } attempt: 5
Sep 25 15:01:25 dist-test-slave-qxh0 krb5kdc[18477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348485, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:01:25.945240 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.928926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:40747 (local address 127.0.0.1:34302)
0925 15:01:25.929231 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:25.929253 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:25.929371 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:25.929944 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:25.929950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:25.930501 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:25.930773 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.930788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.932327 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.932332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.933723 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:25.933738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.934936 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.934942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:25.935167 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.935794 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:25.935816 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:25.938391 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:25.942423 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.942429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.942433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.942981 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.943396 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:25.943402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:25.943405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:25.943563 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:25.944265 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:25.944270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:25.944419 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 15:01:25.944588 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 15:01:25.944866 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260925 15:01:25.946215 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:25.929012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.4.243.252:40747)
0925 15:01:25.929606 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:25.929613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:25.929631 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:01:25.929703 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:25.929709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:25.929787 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:25.929886 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:25.930958 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.932162 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.933889 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:25.934641 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:25.934754 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:25.938650 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:25.938657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:25.938663 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:25.938698 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:25.942261 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.943119 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.943125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.943129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.943272 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:25.943691 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:25.943696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:25.943699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:25.944046 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:25.944202 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:25.945199 (+ 997us) server_negotiation.cc:299] Negotiation successful
0925 15:01:25.945811 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260925 15:01:25.954049 18631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66: Bootstrap starting.
I20260925 15:01:25.957897 18631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:25.959165 18631 log.cc:824] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:25.962387 18631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66: No bootstrap required, opened a new log
I20260925 15:01:25.973662 18631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:25.974112 18631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:25.974231 18631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36349d8837d54b14b01f5b1ad49e6e66, State: Initialized, Role: FOLLOWER
I20260925 15:01:25.974937 18631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:25.976245 18727 raft_consensus.cc:482] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:25.975857 18747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:25.976598 18747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:01:25.976502 18727 raft_consensus.cc:504] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
W20260925 15:01:25.977105 18511 tablet.cc:2369] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:25.977167 18631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:25.978018 18727 leader_election.cc:288] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d82d937710954a23b812a8948d7c2957 (127.4.243.254:35207), 36349d8837d54b14b01f5b1ad49e6e66 (127.4.243.253:32777)
I20260925 15:01:25.979135 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c720f7abf7564ad184c3886e1aca2471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36349d8837d54b14b01f5b1ad49e6e66" is_pre_election: true
I20260925 15:01:25.979637 18606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c720f7abf7564ad184c3886e1aca2471 in term 0.
I20260925 15:01:25.980335 18654 leader_election.cc:302] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36349d8837d54b14b01f5b1ad49e6e66, c720f7abf7564ad184c3886e1aca2471; no voters:
I20260925 15:01:25.980872 18532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c720f7abf7564ad184c3886e1aca2471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82d937710954a23b812a8948d7c2957" is_pre_election: true
I20260925 15:01:25.981447 18532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c720f7abf7564ad184c3886e1aca2471 in term 0.
I20260925 15:01:25.985574 18727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:25.985715 18727 raft_consensus.cc:482] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:25.985801 18727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:25.989355 18727 raft_consensus.cc:504] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:25.990396 18532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c720f7abf7564ad184c3886e1aca2471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82d937710954a23b812a8948d7c2957"
I20260925 15:01:25.990625 18532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:25.993908 18532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c720f7abf7564ad184c3886e1aca2471 in term 1.
I20260925 15:01:25.994369 18655 leader_election.cc:302] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c720f7abf7564ad184c3886e1aca2471, d82d937710954a23b812a8948d7c2957; no voters:
I20260925 15:01:25.997270 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c720f7abf7564ad184c3886e1aca2471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36349d8837d54b14b01f5b1ad49e6e66"
I20260925 15:01:25.997488 18606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:26.000964 18606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c720f7abf7564ad184c3886e1aca2471 in term 1.
I20260925 15:01:26.004998 18727 leader_election.cc:288] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [CANDIDATE]: Term 1 election: Requested vote from peers d82d937710954a23b812a8948d7c2957 (127.4.243.254:35207), 36349d8837d54b14b01f5b1ad49e6e66 (127.4.243.253:32777)
I20260925 15:01:26.005452 18727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:26.005914 18727 raft_consensus.cc:686] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [term 1 LEADER]: Becoming Leader. State: Replica: c720f7abf7564ad184c3886e1aca2471, State: Running, Role: LEADER
I20260925 15:01:26.006738 18727 consensus_queue.cc:237] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } }
I20260925 15:01:26.026309 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c720f7abf7564ad184c3886e1aca2471. Latest consensus state: current_term: 1 leader_uuid: "c720f7abf7564ad184c3886e1aca2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:26.026824 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:26.033885 18758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:26.036218 18758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:26.054558 18606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [term 1 FOLLOWER]: Refusing update from remote peer c720f7abf7564ad184c3886e1aca2471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:26.055531 18751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:26.055778 18762 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:01:26.058568 18532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [term 1 FOLLOWER]: Refusing update from remote peer c720f7abf7564ad184c3886e1aca2471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:26.062042 18751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:26.084952 18747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c720f7abf7564ad184c3886e1aca2471. Latest consensus state: current_term: 1 leader_uuid: "c720f7abf7564ad184c3886e1aca2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:26.085253 18747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:01:26.094017 18760 mvcc.cc:205] Tried to move back new op lower bound from 7333267398868529152 to 7333267398697181184. Current Snapshot: MvccSnapshot[applied={T|T < 7333267398868529152}]
I20260925 15:01:26.099057 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c720f7abf7564ad184c3886e1aca2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:26.099557 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:26.101394 18747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c720f7abf7564ad184c3886e1aca2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:26.101645 18747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:01:26.103888 18758 catalog_manager.cc:1347] Generated new cluster ID: 0fdaca0e182a45a9ad4dd207fee0dbec
I20260925 15:01:26.103995 18758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:26.104189 18712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c720f7abf7564ad184c3886e1aca2471. Latest consensus state: current_term: 1 leader_uuid: "c720f7abf7564ad184c3886e1aca2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:26.104491 18712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:01:26.108314 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c720f7abf7564ad184c3886e1aca2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:26.108547 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:26.120146 18712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c720f7abf7564ad184c3886e1aca2471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82d937710954a23b812a8948d7c2957" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35207 } } peers { permanent_uuid: "36349d8837d54b14b01f5b1ad49e6e66" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32777 } } peers { permanent_uuid: "c720f7abf7564ad184c3886e1aca2471" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 40747 } } }
I20260925 15:01:26.120402 18712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:01:26.130239 18758 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:26.132550 18758 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:26.147743 18758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c720f7abf7564ad184c3886e1aca2471: Generated new TSK 0
I20260925 15:01:26.148773 18758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:01:26.777928 18724 catalog_manager.cc:1259] Loaded cluster ID: 0fdaca0e182a45a9ad4dd207fee0dbec
I20260925 15:01:26.778034 18724 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: loading cluster ID for follower catalog manager: success
I20260925 15:01:26.780386 18724 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: acquiring CA information for follower catalog manager: success
I20260925 15:01:26.782071 18724 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d82d937710954a23b812a8948d7c2957: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:01:27.060271 18762 catalog_manager.cc:1259] Loaded cluster ID: 0fdaca0e182a45a9ad4dd207fee0dbec
I20260925 15:01:27.060405 18762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66: loading cluster ID for follower catalog manager: success
I20260925 15:01:27.062939 18762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66: acquiring CA information for follower catalog manager: success
I20260925 15:01:27.064658 18762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36349d8837d54b14b01f5b1ad49e6e66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:01:27.231240 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18490
I20260925 15:01:27.233132 18552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:27.343860 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18490
I20260925 15:01:27.371485 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18564
I20260925 15:01:27.372648 18626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:27.493422 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18564
I20260925 15:01:27.512998 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18638
I20260925 15:01:27.514158 18700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:27.620396 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18638
2026-09-25T15:01:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4590 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:01:27 dist-test-slave-qxh0 krb5kdc[18781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:01:27 dist-test-slave-qxh0 krb5kdc[18781](info): set up 2 sockets
Sep 25 15:01:27 dist-test-slave-qxh0 krb5kdc[18781](info): commencing operation
krb5kdc: starting...
W20260925 15:01:28.699033 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:01:28 dist-test-slave-qxh0 krb5kdc[18781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:01:28.920606 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:40351
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:29.240885 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:29.241150 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:29.241240 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:29.248353 18794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:29.248467 18794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:29.248514 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:29.248567 18794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:29.248612 18794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:29.258155 18794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:29.259402 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:29.260624 18794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:29.271869 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:29.271827 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:29.271937 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:29.272663 18794 server_base.cc:1034] running on GCE node
I20260925 15:01:29.273379 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:29.274749 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:29.275969 18794 hybrid_clock.cc:648] HybridClock initialized: now 1790348489275923 us; error 50 us; skew 500 ppm
I20260925 15:01:29.276295 18794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:29 dist-test-slave-qxh0 krb5kdc[18781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348489, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:29.279160 18794 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:29.280542 18794 webserver.cc:460] Webserver started at http://127.4.243.254:39215/ using document root <none> and password file <none>
I20260925 15:01:29.281175 18794 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:29.281311 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:29.281621 18794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:29.284659 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "2faf60b0c5874f38a7f7df779eebfccd"
format_stamp: "Formatted at 2026-09-25 15:01:29 on dist-test-slave-qxh0"
server_key: "efef015678e13e38259b932380b36dd5"
server_key_iv: "f563ab8ec0f3f7c58bcf4ef55fc6a310"
server_key_version: "encryptionkey@0"
I20260925 15:01:29.285404 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "2faf60b0c5874f38a7f7df779eebfccd"
format_stamp: "Formatted at 2026-09-25 15:01:29 on dist-test-slave-qxh0"
server_key: "efef015678e13e38259b932380b36dd5"
server_key_iv: "f563ab8ec0f3f7c58bcf4ef55fc6a310"
server_key_version: "encryptionkey@0"
I20260925 15:01:29.290472 18794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:01:29.293792 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:29.295401 18794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:29.295575 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "2faf60b0c5874f38a7f7df779eebfccd"
format_stamp: "Formatted at 2026-09-25 15:01:29 on dist-test-slave-qxh0"
server_key: "efef015678e13e38259b932380b36dd5"
server_key_iv: "f563ab8ec0f3f7c58bcf4ef55fc6a310"
server_key_version: "encryptionkey@0"
I20260925 15:01:29.295766 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:29.319978 18794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:29.323712 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:29.324023 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:29.342353 18794 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:40351
I20260925 15:01:29.342372 18860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:40351 every 8 connection(s)
I20260925 15:01:29.343773 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:29.347462 18861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:29.351737 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18794
I20260925 15:01:29.351956 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:01:29.352771 5071 external_mini_cluster.cc:1428] Setting key c5c52b7c52cb14120fb1b909aa9947ff
I20260925 15:01:29.359107 18861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd: Bootstrap starting.
I20260925 15:01:29.362649 18861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:29.363821 18861 log.cc:824] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:29.367063 18861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd: No bootstrap required, opened a new log
Sep 25 15:01:29 dist-test-slave-qxh0 krb5kdc[18781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:29.375325 18861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf60b0c5874f38a7f7df779eebfccd" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40351 } }
I20260925 15:01:29.375888 18861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:29.376024 18861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2faf60b0c5874f38a7f7df779eebfccd, State: Initialized, Role: FOLLOWER
I20260925 15:01:29.376713 18861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf60b0c5874f38a7f7df779eebfccd" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40351 } }
I20260925 15:01:29.377023 18861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:29.377175 18861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:29.377362 18861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:29.378044 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:29.354772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.4.243.254:40351)
0925 15:01:29.355561 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:29.355573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:29.355613 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:01:29.357794 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:29.357817 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:29.358207 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:29.358578 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:29.360181 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:29.362165 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:29.364231 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:29.365570 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:29.365787 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:29.370274 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:29.370284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:29.370298 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:29.370334 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:29.373508 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:29.374239 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:29.374246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:29.374254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:29.374415 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:29.374886 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:29.374891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:29.374894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:29.375390 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:29.375715 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:29.376164 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 15:01:29.376605 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":356,"threads_started":1}
I20260925 15:01:29.380411 18861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf60b0c5874f38a7f7df779eebfccd" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40351 } }
I20260925 15:01:29.380898 18861 leader_election.cc:302] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2faf60b0c5874f38a7f7df779eebfccd; no voters:
I20260925 15:01:29.381551 18861 leader_election.cc:288] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:29.381691 18866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:29.381982 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:01:29.383564 18861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:29.385900 18866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [term 1 LEADER]: Becoming Leader. State: Replica: 2faf60b0c5874f38a7f7df779eebfccd, State: Running, Role: LEADER
I20260925 15:01:29.386546 18866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf60b0c5874f38a7f7df779eebfccd" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40351 } }
I20260925 15:01:29.389721 18868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2faf60b0c5874f38a7f7df779eebfccd. Latest consensus state: current_term: 1 leader_uuid: "2faf60b0c5874f38a7f7df779eebfccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf60b0c5874f38a7f7df779eebfccd" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40351 } } }
I20260925 15:01:29.390319 18868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:29.390295 18867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2faf60b0c5874f38a7f7df779eebfccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf60b0c5874f38a7f7df779eebfccd" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 40351 } } }
I20260925 15:01:29.390635 18867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd [sys.catalog]: This master's current role is: LEADER
W20260925 15:01:29.398208 18880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:01:29.398344 18880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:01:29.398469 18881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:29.400261 18881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:29.407644 18881 catalog_manager.cc:1347] Generated new cluster ID: 91464cdaa0434177976167d16e931620
I20260925 15:01:29.407757 18881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:29.421752 18881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:29.423527 18881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:29.431059 18881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2faf60b0c5874f38a7f7df779eebfccd: Generated new TSK 0
I20260925 15:01:29.431996 18881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:01:29.443120 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18794
I20260925 15:01:29.444243 18856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:29.550760 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18794
2026-09-25T15:01:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1925 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:01:29 dist-test-slave-qxh0 krb5kdc[18889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:01:29 dist-test-slave-qxh0 krb5kdc[18889](info): set up 2 sockets
Sep 25 15:01:29 dist-test-slave-qxh0 krb5kdc[18889](info): commencing operation
krb5kdc: starting...
W20260925 15:01:30.627236 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:01:30 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:01:30.843894 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36395
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:31.162674 18902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:31.163048 18902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:31.163228 18902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:31.170385 18902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:31.170528 18902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:31.170640 18902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:31.170747 18902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:31.170814 18902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:31.180704 18902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:31.182075 18902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:31.183517 18902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:31.195338 18910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:31.195302 18907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:31.195376 18908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:31.195966 18902 server_base.cc:1034] running on GCE node
I20260925 15:01:31.196739 18902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:31.198167 18902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:31.199425 18902 hybrid_clock.cc:648] HybridClock initialized: now 1790348491199386 us; error 39 us; skew 500 ppm
I20260925 15:01:31.199772 18902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:31 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348491, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:31.209477 18902 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:31.211242 18902 webserver.cc:460] Webserver started at http://127.4.243.254:45161/ using document root <none> and password file <none>
I20260925 15:01:31.212002 18902 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:31.212155 18902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:31.212467 18902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:31.215588 18902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "2d297576003941aa992d9e6e9629baec"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "1ed4b73014951749794d2ae41f11421e"
server_key_iv: "1961be7e9374950e015229cf0b8c7bd3"
server_key_version: "encryptionkey@0"
I20260925 15:01:31.216325 18902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "2d297576003941aa992d9e6e9629baec"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "1ed4b73014951749794d2ae41f11421e"
server_key_iv: "1961be7e9374950e015229cf0b8c7bd3"
server_key_version: "encryptionkey@0"
I20260925 15:01:31.221414 18902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:01:31.224766 18917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:31.226034 18902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 15:01:31.226227 18902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "2d297576003941aa992d9e6e9629baec"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "1ed4b73014951749794d2ae41f11421e"
server_key_iv: "1961be7e9374950e015229cf0b8c7bd3"
server_key_version: "encryptionkey@0"
I20260925 15:01:31.226433 18902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:31.242357 18902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:31.246485 18902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:31.246809 18902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:31.265084 18902 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:36395
I20260925 15:01:31.265131 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:36395 every 8 connection(s)
I20260925 15:01:31.266448 18902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:31.270033 18969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:31.275128 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18902
I20260925 15:01:31.275342 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:01:31.276122 5071 external_mini_cluster.cc:1428] Setting key 34fe9d1a3ebf3d63536700ce353b6834
I20260925 15:01:31.281059 18969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Bootstrap starting.
I20260925 15:01:31.284061 18969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:31.285048 18969 log.cc:824] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:31.288058 18969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: No bootstrap required, opened a new log
Sep 25 15:01:31 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:31.298264 18969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:31.298738 18969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:31.298849 18969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d297576003941aa992d9e6e9629baec, State: Initialized, Role: FOLLOWER
I20260925 15:01:31.299173 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:31.278183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.4.243.254:36395)
0925 15:01:31.278989 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:31.279000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:31.279039 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:01:31.280575 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:31.280595 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:31.280960 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:31.281317 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:31.282729 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:31.283999 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:31.285316 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:31.286268 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:31.286459 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:31.290556 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:31.290572 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:31.290592 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:31.290660 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:31.294618 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:31.295270 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:31.295278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:31.295287 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:31.295474 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:31.295862 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:31.295868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:31.295871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:31.296628 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:31.296939 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:31.297294 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:01:31.297739 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":403,"threads_started":1}
I20260925 15:01:31.299494 18969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:31.299760 18969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:31.299914 18969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:31.300107 18969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:31.302305 18969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:31.302881 18969 leader_election.cc:302] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d297576003941aa992d9e6e9629baec; no voters:
I20260925 15:01:31.303442 18969 leader_election.cc:288] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:31.303594 18974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:31.304342 18974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 LEADER]: Becoming Leader. State: Replica: 2d297576003941aa992d9e6e9629baec, State: Running, Role: LEADER
I20260925 15:01:31.304860 18974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:31.305565 18969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:31.307602 18976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d297576003941aa992d9e6e9629baec. Latest consensus state: current_term: 1 leader_uuid: "2d297576003941aa992d9e6e9629baec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } } }
I20260925 15:01:31.308032 18976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:31.308586 18975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d297576003941aa992d9e6e9629baec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } } }
I20260925 15:01:31.308939 18975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:31.311276 18983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:31.315549 18983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:31.324204 18983 catalog_manager.cc:1347] Generated new cluster ID: 5b48b18132e04a24a75ebfc1c7491a35
I20260925 15:01:31.324366 18983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:31.335088 18983 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:31.336346 18983 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:31.343369 18983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Generated new TSK 0
I20260925 15:01:31.344158 18983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 15:01:31.397989 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36395
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:31.716920 18995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:31.717177 18995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:31.717268 18995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:31.724135 18995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:31.724243 18995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:31.724426 18995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:01:31.734529 18995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:31.735875 18995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:31.737185 18995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:31.749557 19000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:31.749522 19003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:31.749600 19001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:31.750742 18995 server_base.cc:1034] running on GCE node
I20260925 15:01:31.751328 18995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:31.752236 18995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:31.753436 18995 hybrid_clock.cc:648] HybridClock initialized: now 1790348491753401 us; error 24 us; skew 500 ppm
I20260925 15:01:31.753690 18995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:31 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348491, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:31.756630 18995 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:01:31.757978 18995 webserver.cc:460] Webserver started at http://127.4.243.193:33069/ using document root <none> and password file <none>
I20260925 15:01:31.758690 18995 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:31.758790 18995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:31.759037 18995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:31.761684 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "ccccd14d392b43e39f8b5213d9815e65"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "4ad3e04eefb4525b916ad1df1468016e"
server_key_iv: "84c5efed8987a7d25e7c9554132161c5"
server_key_version: "encryptionkey@0"
I20260925 15:01:31.762346 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "ccccd14d392b43e39f8b5213d9815e65"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "4ad3e04eefb4525b916ad1df1468016e"
server_key_iv: "84c5efed8987a7d25e7c9554132161c5"
server_key_version: "encryptionkey@0"
I20260925 15:01:31.767303 18995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 15:01:31.770937 19010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:31.772349 18995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:01:31.772532 18995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "ccccd14d392b43e39f8b5213d9815e65"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "4ad3e04eefb4525b916ad1df1468016e"
server_key_iv: "84c5efed8987a7d25e7c9554132161c5"
server_key_version: "encryptionkey@0"
I20260925 15:01:31.772734 18995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:31.790834 18995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:31.794973 18995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:31.795234 18995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:31.796233 18995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:31.797605 18995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:31.797704 18995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:31.797847 18995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:31.797924 18995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:31.831349 18995 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:42905
I20260925 15:01:31.831393 19122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:42905 every 8 connection(s)
I20260925 15:01:31.832609 18995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:31.839905 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 18995
I20260925 15:01:31.840119 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:31.840720 5071 external_mini_cluster.cc:1428] Setting key 60f9ca64c59e7871bb40fbf53e422b44
Sep 25 15:01:31 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348491, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:31.853786 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:31.835052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:52791 (local address 127.4.243.254:36395)
0925 15:01:31.835537 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:31.835542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:31.836492 (+ 950us) server_negotiation.cc:407] Connection header received
0925 15:01:31.838132 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:31.838137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:31.838199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:31.838293 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:31.839877 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:31.840695 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:31.842725 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:31.843329 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:31.843504 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:31.847124 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:31.847133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:31.847141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:31.847188 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:31.850445 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:31.851448 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:31.851452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:31.851454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:31.851588 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:31.852085 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:31.852088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:31.852090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:31.852363 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:31.852472 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:31.853354 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 15:01:31.853520 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":251,"threads_started":1}
I20260925 15:01:31.855391 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:31.835664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:52791)
0925 15:01:31.836320 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:31.836360 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:31.837901 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:31.838482 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:31.838489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:31.839050 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:31.839704 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:31.839729 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:31.840854 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:31.840861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:31.842557 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:31.842568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:31.843588 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:31.843598 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:31.843810 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:31.844493 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:31.844520 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:31.846861 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:31.850618 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:31.850630 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:31.850652 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:31.851305 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:31.851701 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:31.851708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:31.851714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:31.851970 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:31.852489 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:31.852500 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:31.852970 (+ 470us) client_negotiation.cc:769] Sending connection context
0925 15:01:31.853269 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 15:01:31.853873 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":261,"threads_started":1}
I20260925 15:01:31.857015 19123 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36395
I20260925 15:01:31.857318 19123 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:31.858040 19123 heartbeater.cc:511] Master 127.4.243.254:36395 requested a full tablet report, sending...
I20260925 15:01:31.860178 18934 ts_manager.cc:194] Registered new tserver with Master: ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:31.861661 18934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:52791
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 15:01:31.894398 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36395
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:32.213796 19129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:32.214118 19129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:32.214289 19129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:32.221287 19129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:32.221423 19129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:32.221624 19129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:01:32.231621 19129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:32.232867 19129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:32.234138 19129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:32.246891 19134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:32.246886 19137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:32.246806 19135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:32.248035 19129 server_base.cc:1034] running on GCE node
I20260925 15:01:32.248637 19129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:32.249603 19129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:32.250831 19129 hybrid_clock.cc:648] HybridClock initialized: now 1790348492250742 us; error 78 us; skew 500 ppm
I20260925 15:01:32.251168 19129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:32 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348492, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:32.254352 19129 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:01:32.255838 19129 webserver.cc:460] Webserver started at http://127.4.243.194:38509/ using document root <none> and password file <none>
I20260925 15:01:32.256472 19129 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:32.256567 19129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:32.256803 19129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:32.259776 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "d9af8329ab0b41209ff9163d6104fa60"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "5a0c0e164498a9a18a00e011ed7137ca"
server_key_iv: "0b3e78c14a0dc74ce1fd82d772a44985"
server_key_version: "encryptionkey@0"
I20260925 15:01:32.260452 19129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "d9af8329ab0b41209ff9163d6104fa60"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "5a0c0e164498a9a18a00e011ed7137ca"
server_key_iv: "0b3e78c14a0dc74ce1fd82d772a44985"
server_key_version: "encryptionkey@0"
I20260925 15:01:32.265542 19129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:01:32.268913 19144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:32.270488 19129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:01:32.270689 19129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "d9af8329ab0b41209ff9163d6104fa60"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "5a0c0e164498a9a18a00e011ed7137ca"
server_key_iv: "0b3e78c14a0dc74ce1fd82d772a44985"
server_key_version: "encryptionkey@0"
I20260925 15:01:32.270913 19129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:32.280970 19129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:32.285007 19129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:32.285272 19129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:32.286180 19129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:32.287571 19129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:32.287642 19129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:32.287724 19129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:32.287765 19129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:32.320386 19129 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:32843
I20260925 15:01:32.320420 19256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:32843 every 8 connection(s)
I20260925 15:01:32.321623 19129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:32.326136 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 19129
I20260925 15:01:32.326345 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:32.326933 5071 external_mini_cluster.cc:1428] Setting key 7026243c6eb2838ba02aca3bc75b1de0
Sep 25 15:01:32 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348492, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:32.344347 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.324110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:56665 (local address 127.4.243.254:36395)
0925 15:01:32.324286 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.324290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.325494 (+ 1204us) server_negotiation.cc:407] Connection header received
0925 15:01:32.328001 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.328007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.328102 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.328221 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:32.330079 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.331029 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.333572 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.333986 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.334081 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.338420 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:32.338425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:32.338428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:32.338460 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:32.341003 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:32.341901 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:32.341904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:32.341907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:32.342041 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:32.342496 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:32.342499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:32.342501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:32.342795 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:32.342904 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.343928 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.344115 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:01:32.345831 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.324665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:56665)
0925 15:01:32.325336 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:32.325371 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:32.327706 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:32.328468 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:32.328479 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:32.329065 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:32.329889 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.329919 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.331180 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.331188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.333397 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.333416 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.334219 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.334229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.334555 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.335350 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:32.335385 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:32.338205 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:32.341136 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:32.341148 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:32.341168 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:32.341778 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:32.342144 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:32.342152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:32.342159 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:32.342389 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:32.342918 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:32.342929 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:32.343432 (+ 503us) client_negotiation.cc:769] Sending connection context
0925 15:01:32.343753 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 15:01:32.344353 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":264,"threads_started":1}
I20260925 15:01:32.347374 19257 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36395
I20260925 15:01:32.347685 19257 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:32.348435 19257 heartbeater.cc:511] Master 127.4.243.254:36395 requested a full tablet report, sending...
I20260925 15:01:32.349962 18934 ts_manager.cc:194] Registered new tserver with Master: d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843)
I20260925 15:01:32.350786 18934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:56665
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 15:01:32.379994 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36395
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:32.706314 19262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:32.706574 19262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:32.706791 19262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:32.714097 19262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:32.714187 19262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:32.714350 19262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:01:32.723449 19262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:32.724665 19262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:32.725967 19262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:32.742983 19270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:32.743031 19268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:32.743057 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:32.744611 19262 server_base.cc:1034] running on GCE node
I20260925 15:01:32.745178 19262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:32.746178 19262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:32.747370 19262 hybrid_clock.cc:648] HybridClock initialized: now 1790348492747330 us; error 27 us; skew 500 ppm
I20260925 15:01:32.747690 19262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:32 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348492, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:32.750554 19262 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:01:32.751997 19262 webserver.cc:460] Webserver started at http://127.4.243.195:45157/ using document root <none> and password file <none>
I20260925 15:01:32.752684 19262 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:32.752825 19262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:32.753125 19262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:32.755908 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "be8eff67756749faacd8187d16dce431"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "1e1043815f9a747b40455942ff711d5a"
server_key_iv: "f6368d8112b89f7df95f5a2d561a5d47"
server_key_version: "encryptionkey@0"
I20260925 15:01:32.756631 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "be8eff67756749faacd8187d16dce431"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "1e1043815f9a747b40455942ff711d5a"
server_key_iv: "f6368d8112b89f7df95f5a2d561a5d47"
server_key_version: "encryptionkey@0"
I20260925 15:01:32.761399 19262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:01:32.764689 19277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:32.765985 19262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:01:32.766166 19262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "be8eff67756749faacd8187d16dce431"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "1e1043815f9a747b40455942ff711d5a"
server_key_iv: "f6368d8112b89f7df95f5a2d561a5d47"
server_key_version: "encryptionkey@0"
I20260925 15:01:32.766420 19262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:32.797103 19262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:32.801049 19262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:32.801330 19262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:32.802276 19262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:32.803795 19262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:32.803905 19262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:32.804050 19262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:32.804119 19262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:32.839290 19262 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:34517
I20260925 15:01:32.839344 19389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:34517 every 8 connection(s)
I20260925 15:01:32.840502 19262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:32.842932 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 19262
I20260925 15:01:32.843127 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:32.843652 5071 external_mini_cluster.cc:1428] Setting key 343a69ab75b05e516a6f7368d55b3770
Sep 25 15:01:32 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348492, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:32.860320 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.843026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:57557 (local address 127.4.243.254:36395)
0925 15:01:32.843267 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.843271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.844691 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 15:01:32.847121 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.847126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.847249 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.847359 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:32.848854 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.849610 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.851235 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.851635 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.851774 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.855084 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:32.855089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:32.855092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:32.855123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:32.857387 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:32.858208 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:32.858213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:32.858219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:32.858336 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:32.858819 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:32.858823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:32.858825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:32.859193 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:32.859306 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.859922 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.860085 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:01:32.861235 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.843631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:57557)
0925 15:01:32.844537 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:32.844580 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:32.846833 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:32.847607 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:32.847615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:32.848134 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:32.848688 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.848704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.849760 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.849770 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.851088 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.851099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.851863 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.851871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.852118 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.852738 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:32.852760 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:32.854871 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:32.857531 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:32.857539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:32.857555 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:32.858086 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:32.858443 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:32.858449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:32.858453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:32.858691 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:32.859317 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:32.859325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:32.859621 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 15:01:32.859834 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 15:01:32.860198 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":366,"threads_started":1}
I20260925 15:01:32.862473 19390 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36395
I20260925 15:01:32.862788 19390 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:32.863456 19390 heartbeater.cc:511] Master 127.4.243.254:36395 requested a full tablet report, sending...
I20260925 15:01:32.864843 18934 ts_manager.cc:194] Registered new tserver with Master: be8eff67756749faacd8187d16dce431 (127.4.243.195:34517)
I20260925 15:01:32.865003 19123 heartbeater.cc:503] Master 127.4.243.254:36395 was elected leader, sending a full tablet report...
I20260925 15:01:32.865825 18934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:57557
I20260925 15:01:32.871281 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:32.886920 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.875518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.4.243.254:36395)
0925 15:01:32.875708 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.875717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.876088 (+ 371us) server_negotiation.cc:407] Connection header received
0925 15:01:32.876231 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.876235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.876288 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.876401 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:32.877739 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.878416 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.879620 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.879993 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.880078 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.882187 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:32.882191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:32.882194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:32.882221 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:32.884648 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:32.885194 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:32.885197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:32.885199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:32.885317 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:32.885604 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:32.885606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:32.885608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:32.885941 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:32.886114 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.886395 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.886546 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:01:32.890790 18934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:32.897286 18934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:32.919194 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.908333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:34517 (local address 127.0.0.1:38730)
0925 15:01:32.909074 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:32.909113 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:32.909291 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:32.911736 (+ 2445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:32.911745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:32.911779 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:32.912237 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.912249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.914313 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.914320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.916458 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.916476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.918182 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.918188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.918376 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.918428 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:01:32.918532 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:01:32.918736 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":38,"spinlock_wait_cycles":9472,"thread_start_us":338,"threads_started":1}
I20260925 15:01:32.920547 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.908437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.4.243.195:34517)
0925 15:01:32.910996 (+ 2559us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.911008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.911045 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:01:32.911158 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.911168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.911492 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.911706 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:32.912588 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.914195 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.916630 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.917989 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.919786 (+ 1797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.920004 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.920073 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.920208 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2350,"thread_start_us":2175,"threads_started":1}
I20260925 15:01:32.920673 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.908840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59068)
0925 15:01:32.910196 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:32.910214 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:32.910313 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:32.912920 (+ 2607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:32.912925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:32.912948 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:32.913421 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.913436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.915299 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.915304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.917607 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.917621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.919896 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.919902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.920059 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.920089 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:01:32.920191 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:01:32.920306 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":109,"thread_start_us":1097,"threads_started":1}
I20260925 15:01:32.922632 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.909455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.4.243.193:42905)
0925 15:01:32.912402 (+ 2947us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.912410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.912439 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:01:32.912528 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.912535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.912745 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.912935 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:32.913596 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.915165 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.917808 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.919719 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.919855 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.920084 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.920221 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.920360 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2776,"thread_start_us":2625,"threads_started":1}
I20260925 15:01:32.924639 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.909599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:32843 (local address 127.0.0.1:36512)
0925 15:01:32.911700 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:32.911715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:32.911825 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:32.913219 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:32.913226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:32.913251 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:32.913676 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.913687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.915488 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.915493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.917822 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.917835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.923871 (+ 6036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.923877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.924033 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.924067 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:01:32.924156 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:01:32.924272 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1967,"mutex_wait_us":45,"spinlock_wait_cycles":384,"thread_start_us":1850,"threads_started":1}
I20260925 15:01:32.924860 19325 tablet_service.cc:1467] Processing CreateTablet for tablet 67a6baacf6e646de8e37e9072b38c7db (DEFAULT_TABLE table=test-table [id=b70aeb1460eb4787b16a6573845d2902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:32.925505 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.910165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.4.243.194:32843)
0925 15:01:32.911280 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.911291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.912048 (+ 757us) server_negotiation.cc:407] Connection header received
0925 15:01:32.912157 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.912163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.912403 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.912590 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:32.913838 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.915355 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.918013 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.923428 (+ 5415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.923561 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.923728 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.925059 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.925219 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":776,"threads_started":1}
I20260925 15:01:32.926395 19325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a6baacf6e646de8e37e9072b38c7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:32.929313 19192 tablet_service.cc:1467] Processing CreateTablet for tablet 67a6baacf6e646de8e37e9072b38c7db (DEFAULT_TABLE table=test-table [id=b70aeb1460eb4787b16a6573845d2902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:32.930723 19192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a6baacf6e646de8e37e9072b38c7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:32.929584 19058 tablet_service.cc:1467] Processing CreateTablet for tablet 67a6baacf6e646de8e37e9072b38c7db (DEFAULT_TABLE table=test-table [id=b70aeb1460eb4787b16a6573845d2902]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:32.931897 19058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a6baacf6e646de8e37e9072b38c7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:32.941977 19408 tablet_bootstrap.cc:492] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Bootstrap starting.
I20260925 15:01:32.944775 19409 tablet_bootstrap.cc:492] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Bootstrap starting.
I20260925 15:01:32.945501 19408 tablet_bootstrap.cc:654] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:32.946772 19408 log.cc:824] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:32.948014 19409 tablet_bootstrap.cc:654] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:32.948860 19411 tablet_bootstrap.cc:492] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Bootstrap starting.
I20260925 15:01:32.949270 19409 log.cc:824] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:32.949855 19408 tablet_bootstrap.cc:492] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: No bootstrap required, opened a new log
I20260925 15:01:32.950138 19408 ts_tablet_manager.cc:1397] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:01:32.952178 19409 tablet_bootstrap.cc:492] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: No bootstrap required, opened a new log
I20260925 15:01:32.952196 19411 tablet_bootstrap.cc:654] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:32.952535 19409 ts_tablet_manager.cc:1397] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:01:32.953330 19411 log.cc:824] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:32.956099 19411 tablet_bootstrap.cc:492] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: No bootstrap required, opened a new log
I20260925 15:01:32.956341 19411 ts_tablet_manager.cc:1397] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 15:01:32.957860 19408 raft_consensus.cc:348] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.958199 19408 raft_consensus.cc:374] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:32.958288 19408 raft_consensus.cc:729] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8eff67756749faacd8187d16dce431, State: Initialized, Role: FOLLOWER
I20260925 15:01:32.959661 19408 consensus_queue.cc:260] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.961007 19390 heartbeater.cc:503] Master 127.4.243.254:36395 was elected leader, sending a full tablet report...
I20260925 15:01:32.962495 19409 raft_consensus.cc:348] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.962968 19409 raft_consensus.cc:374] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:32.963058 19409 raft_consensus.cc:729] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9af8329ab0b41209ff9163d6104fa60, State: Initialized, Role: FOLLOWER
I20260925 15:01:32.963548 19408 ts_tablet_manager.cc:1428] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260925 15:01:32.963717 19409 consensus_queue.cc:260] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.965251 19257 heartbeater.cc:503] Master 127.4.243.254:36395 was elected leader, sending a full tablet report...
I20260925 15:01:32.966580 19411 raft_consensus.cc:348] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.967083 19411 raft_consensus.cc:374] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:32.967185 19411 raft_consensus.cc:729] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccccd14d392b43e39f8b5213d9815e65, State: Initialized, Role: FOLLOWER
I20260925 15:01:32.967722 19409 ts_tablet_manager.cc:1428] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260925 15:01:32.967846 19414 raft_consensus.cc:482] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:32.967830 19411 consensus_queue.cc:260] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.968127 19414 raft_consensus.cc:504] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.969741 19411 ts_tablet_manager.cc:1428] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260925 15:01:32.971838 19414 leader_election.cc:288] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843), ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:32.978005 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.970294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:32843 (local address 127.4.243.195:51205)
0925 15:01:32.970500 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:32.970542 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:32.970689 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:32.971644 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:32.971650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:32.971683 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:32.971920 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.971938 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.973404 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.973410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.975563 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.975576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.977265 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.977271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.977451 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.977487 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:01:32.977573 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:01:32.977709 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260925 15:01:32.978672 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.970858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:51205 (local address 127.4.243.194:32843)
0925 15:01:32.971179 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.971185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.971202 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:01:32.971268 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.971273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.971344 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.971442 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:32.972089 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.973198 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.975733 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.976963 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.977112 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.977185 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.978250 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.978387 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260925 15:01:32.979087 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.971186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:47231 (local address 127.4.243.193:42905)
0925 15:01:32.971365 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:32.971372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:32.971996 (+ 624us) server_negotiation.cc:407] Connection header received
0925 15:01:32.972291 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:32.972297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:32.972383 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:32.972445 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:32.973187 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.974273 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.976748 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.977957 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.978090 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.978168 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:32.978754 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 15:01:32.978887 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:01:32.979238 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:32.971178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.4.243.195:47231)
0925 15:01:32.971893 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:32.971917 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:32.972145 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:32.972664 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:32.972669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:32.972693 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:32.973012 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.973022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.974497 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.974502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.976581 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:32.976597 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:32.978369 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:32.978374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:32.978553 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:32.978583 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:01:32.978728 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:01:32.978865 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":22,"thread_start_us":358,"threads_started":1}
I20260925 15:01:32.979568 19212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6baacf6e646de8e37e9072b38c7db" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9af8329ab0b41209ff9163d6104fa60" is_pre_election: true
I20260925 15:01:32.980036 19212 raft_consensus.cc:2454] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8eff67756749faacd8187d16dce431 in term 0.
I20260925 15:01:32.980654 19281 leader_election.cc:302] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8eff67756749faacd8187d16dce431, d9af8329ab0b41209ff9163d6104fa60; no voters:
I20260925 15:01:32.981014 19414 raft_consensus.cc:2781] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:32.981165 19414 raft_consensus.cc:482] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:32.981277 19414 raft_consensus.cc:3037] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:32.981197 19078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6baacf6e646de8e37e9072b38c7db" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccccd14d392b43e39f8b5213d9815e65" is_pre_election: true
I20260925 15:01:32.981674 19078 raft_consensus.cc:2454] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8eff67756749faacd8187d16dce431 in term 0.
I20260925 15:01:32.983743 19414 raft_consensus.cc:504] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.984314 19414 leader_election.cc:288] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 election: Requested vote from peers d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843), ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:32.984735 19078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6baacf6e646de8e37e9072b38c7db" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccccd14d392b43e39f8b5213d9815e65"
I20260925 15:01:32.984710 19212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a6baacf6e646de8e37e9072b38c7db" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9af8329ab0b41209ff9163d6104fa60"
I20260925 15:01:32.984903 19078 raft_consensus.cc:3037] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:32.984903 19212 raft_consensus.cc:3037] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:32.988080 19078 raft_consensus.cc:2454] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8eff67756749faacd8187d16dce431 in term 1.
I20260925 15:01:32.988080 19212 raft_consensus.cc:2454] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8eff67756749faacd8187d16dce431 in term 1.
I20260925 15:01:32.988474 19279 leader_election.cc:302] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8eff67756749faacd8187d16dce431, ccccd14d392b43e39f8b5213d9815e65; no voters:
I20260925 15:01:32.988816 19414 raft_consensus.cc:2781] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:32.989255 19414 raft_consensus.cc:686] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 1 LEADER]: Becoming Leader. State: Replica: be8eff67756749faacd8187d16dce431, State: Running, Role: LEADER
I20260925 15:01:32.989699 19414 consensus_queue.cc:237] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:32.994134 18934 catalog_manager.cc:5495] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 reported cstate change: term changed from 0 to 1, leader changed from <none> to be8eff67756749faacd8187d16dce431 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "be8eff67756749faacd8187d16dce431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } health_report { overall_health: UNKNOWN } } }
I20260925 15:01:33.024729 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.019641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.4.243.195:34517)
0925 15:01:33.019866 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:33.019874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:33.019898 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:01:33.020006 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:33.020009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:33.020098 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:33.020211 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:33.020590 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.021300 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.022717 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.023123 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.023229 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:33.023672 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:33.024181 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:33.024289 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:33.024373 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:01:33.024436 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:01:33.032943 19078 raft_consensus.cc:1261] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Refusing update from remote peer be8eff67756749faacd8187d16dce431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:33.032959 19212 raft_consensus.cc:1261] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Refusing update from remote peer be8eff67756749faacd8187d16dce431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:33.034024 19418 consensus_queue.cc:1035] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:33.034415 19414 consensus_queue.cc:1035] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:33.046237 19421 mvcc.cc:205] Tried to move back new op lower bound from 7333267427451650048 to 7333267427289706496. Current Snapshot: MvccSnapshot[applied={T|T < 7333267427451650048}]
I20260925 15:01:33.050773 19423 mvcc.cc:205] Tried to move back new op lower bound from 7333267427451650048 to 7333267427289706496. Current Snapshot: MvccSnapshot[applied={T|T < 7333267427451650048}]
I20260925 15:01:33.061203 18933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:33.061468 18933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:33.065017 18933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Sending DeleteTablet for 3 replicas of tablet 67a6baacf6e646de8e37e9072b38c7db
I20260925 15:01:33.066054 19325 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a6baacf6e646de8e37e9072b38c7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:33 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:38730
I20260925 15:01:33.066165 19192 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a6baacf6e646de8e37e9072b38c7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:33 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:36512
I20260925 15:01:33.066668 19058 tablet_service.cc:1514] Processing DeleteTablet for tablet 67a6baacf6e646de8e37e9072b38c7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:33 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:59068
I20260925 15:01:33.066982 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18995
I20260925 15:01:33.076156 19432 tablet_replica.cc:330] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: stopping tablet replica
I20260925 15:01:33.077221 19432 raft_consensus.cc:2229] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:33.078197 19432 raft_consensus.cc:2258] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:33.080596 19432 ts_tablet_manager.cc:1905] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:33.083142 19433 tablet_replica.cc:330] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: stopping tablet replica
I20260925 15:01:33.083686 19431 tablet_replica.cc:330] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: stopping tablet replica
I20260925 15:01:33.084192 19431 raft_consensus.cc:2229] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:33.084672 19433 raft_consensus.cc:2229] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:33.085204 19433 raft_consensus.cc:2258] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:33.085819 19431 raft_consensus.cc:2258] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:33.087886 19432 ts_tablet_manager.cc:1918] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:33.088052 19432 log.cc:1196] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/67a6baacf6e646de8e37e9072b38c7db
I20260925 15:01:33.088635 19432 ts_tablet_manager.cc:1939] T 67a6baacf6e646de8e37e9072b38c7db P d9af8329ab0b41209ff9163d6104fa60: Deleting consensus metadata
Sep 25 15:01:33 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 15:01:33.090072 19431 ts_tablet_manager.cc:1905] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:33.090466 19433 ts_tablet_manager.cc:1905] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:33.090469 18921 catalog_manager.cc:4841] TS d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843): tablet 67a6baacf6e646de8e37e9072b38c7db (table test-table [id=b70aeb1460eb4787b16a6573845d2902]) successfully deleted
I20260925 15:01:33.097961 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.068285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.4.243.193:42905)
0925 15:01:33.068503 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:33.068511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:33.076544 (+ 8033us) server_negotiation.cc:407] Connection header received
0925 15:01:33.076838 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:33.076866 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:33.076987 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:33.077057 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:33.079571 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.080722 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.083572 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.084241 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.084340 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:33.090406 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:33.090421 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:33.090455 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:33.090502 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:33.094425 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:33.095395 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:33.095404 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:33.095408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:33.095572 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:33.096018 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:33.096022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:33.096025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:33.096514 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:33.096792 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:33.097338 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 15:01:33.097586 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:01:33.098268 19431 ts_tablet_manager.cc:1918] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:33.097610 19433 ts_tablet_manager.cc:1918] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:33.098433 19431 log.cc:1196] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/67a6baacf6e646de8e37e9072b38c7db
I20260925 15:01:33.098552 19433 log.cc:1196] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/67a6baacf6e646de8e37e9072b38c7db
I20260925 15:01:33.098944 19118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:33.099035 19431 ts_tablet_manager.cc:1939] T 67a6baacf6e646de8e37e9072b38c7db P be8eff67756749faacd8187d16dce431: Deleting consensus metadata
I20260925 15:01:33.100678 18921 catalog_manager.cc:4841] TS be8eff67756749faacd8187d16dce431 (127.4.243.195:34517): tablet 67a6baacf6e646de8e37e9072b38c7db (table test-table [id=b70aeb1460eb4787b16a6573845d2902]) successfully deleted
I20260925 15:01:33.204730 19433 ts_tablet_manager.cc:1939] T 67a6baacf6e646de8e37e9072b38c7db P ccccd14d392b43e39f8b5213d9815e65: Deleting consensus metadata
I20260925 15:01:33.205519 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18995
I20260925 15:01:33.223071 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 19129
W20260925 15:01:33.223110 18919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:42905 (error 108)
I20260925 15:01:33.224980 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.224533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59086)
0925 15:01:33.224674 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:33.224762 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:01:33.225208 18919 catalog_manager.cc:4584] TS ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67a6baacf6e646de8e37e9072b38c7db: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Sep 25 15:01:33 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:01:33.241451 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.224232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.4.243.194:32843)
0925 15:01:33.224418 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:33.224424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:33.224859 (+ 435us) server_negotiation.cc:407] Connection header received
0925 15:01:33.225041 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:33.225047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:33.225148 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:33.225257 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:33.227045 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.227775 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.229652 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.230065 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.230229 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:33.235214 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:33.235231 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:33.235249 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:33.235290 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:33.238815 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:33.239429 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:33.239438 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:33.239441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:33.239606 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:33.239930 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:33.239938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:33.239942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:33.240397 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:33.240680 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:33.240863 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:01:33.241076 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:01:33.242226 19252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:33.279045 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.278530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59092)
0925 15:01:33.278742 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:33.278851 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 15:01:33.280246 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.279852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59096)
0925 15:01:33.280021 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:33.280113 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 15:01:33.341471 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.340388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59100)
0925 15:01:33.341126 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:33.341250 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576}
I20260925 15:01:33.342789 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.342284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59110)
0925 15:01:33.342526 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:33.342656 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 15:01:33.347256 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 19129
I20260925 15:01:33.364482 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 19262
Sep 25 15:01:33 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 15:01:33.382326 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.365279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.4.243.195:34517)
0925 15:01:33.365490 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:33.365498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:33.365520 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:01:33.365594 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:33.365603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:33.365695 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:33.365825 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:33.367131 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.368448 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.369815 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:33.370571 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:33.370891 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:33.375275 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:33.375289 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:33.375317 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:33.375369 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:33.379241 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:33.380111 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:33.380125 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:33.380130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:33.380342 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:33.380719 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:33.380724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:33.380727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:33.381149 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:33.381491 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:33.381654 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:01:33.381874 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:01:33.383085 19385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:33.423725 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.423257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59112)
0925 15:01:33.423474 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:33.423570 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:01:33.424831 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:33.424443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:59116)
0925 15:01:33.424644 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:33.424718 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 15:01:33.506712 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 19262
I20260925 15:01:33.527734 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 18902
I20260925 15:01:33.528996 18964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:33.638579 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 18902
I20260925 15:01:33.660063 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36395
--webserver_interface=127.4.243.254
--webserver_port=45161
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:33.982446 19440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:33.982728 19440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:33.982911 19440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:33.989758 19440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:33.989883 19440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:33.989967 19440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:33.990067 19440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:33.990144 19440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:33.999274 19440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=45161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:34.000427 19440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:34.001652 19440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:34.023692 19447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:34.023692 19446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:34.023763 19449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:34.024329 19440 server_base.cc:1034] running on GCE node
I20260925 15:01:34.025172 19440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:34.026798 19440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:34.028007 19440 hybrid_clock.cc:648] HybridClock initialized: now 1790348494027957 us; error 31 us; skew 500 ppm
I20260925 15:01:34.028391 19440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:34 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348494, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:34.031876 19440 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:34.033437 19440 webserver.cc:460] Webserver started at http://127.4.243.254:45161/ using document root <none> and password file <none>
I20260925 15:01:34.034197 19440 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:34.034345 19440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:34.040648 19440 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:01:34.043429 19456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:34.044775 19440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:34.044955 19440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "2d297576003941aa992d9e6e9629baec"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "1ed4b73014951749794d2ae41f11421e"
server_key_iv: "1961be7e9374950e015229cf0b8c7bd3"
server_key_version: "encryptionkey@0"
I20260925 15:01:34.046042 19440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:34.061866 19440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:34.065907 19440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:34.066205 19440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:34.084650 19440 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:36395
I20260925 15:01:34.084661 19507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:36395 every 8 connection(s)
I20260925 15:01:34.086226 19440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:34.091506 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 19440
I20260925 15:01:34.092306 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:42905
--local_ip_for_outbound_sockets=127.4.243.193
--tserver_master_addrs=127.4.243.254:36395
--webserver_port=33069
--webserver_interface=127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:34.093572 19508 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:01:34.096916 19508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Bootstrap starting.
I20260925 15:01:34.124485 19508 log.cc:824] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:34.140193 19508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:01:34.140821 19508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Bootstrap complete.
I20260925 15:01:34.154484 19508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:34.155973 19508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d297576003941aa992d9e6e9629baec, State: Initialized, Role: FOLLOWER
I20260925 15:01:34.156790 19508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:34.157126 19508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:34.157279 19508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:34.157501 19508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:01:34.163393 19508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:34.163990 19508 leader_election.cc:302] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d297576003941aa992d9e6e9629baec; no voters:
I20260925 15:01:34.164621 19508 leader_election.cc:288] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:01:34.164776 19512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:01:34.165961 19512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 LEADER]: Becoming Leader. State: Replica: 2d297576003941aa992d9e6e9629baec, State: Running, Role: LEADER
I20260925 15:01:34.166517 19512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:34.166843 19508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:34.169018 19514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d297576003941aa992d9e6e9629baec. Latest consensus state: current_term: 2 leader_uuid: "2d297576003941aa992d9e6e9629baec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } } }
I20260925 15:01:34.169418 19514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:34.169950 19513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2d297576003941aa992d9e6e9629baec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } } }
I20260925 15:01:34.170271 19513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:34.180449 19518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:34.188428 19518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:34.189810 19518 catalog_manager.cc:1259] Loaded cluster ID: 5b48b18132e04a24a75ebfc1c7491a35
I20260925 15:01:34.189914 19518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:34.192955 19518 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:01:34.210945 19518 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:01:34.220042 19518 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:34.221715 19518 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Loaded TSK: 0
I20260925 15:01:34.223222 19518 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:01:34.457311 19510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:34.457568 19510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:34.457710 19510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:34.464792 19510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:34.464885 19510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:34.465090 19510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:01:34.475044 19510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:42905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=33069
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:34.476322 19510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:34.477613 19510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:34.492206 19535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:34.492164 19536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:34.493058 19538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:34.493718 19510 server_base.cc:1034] running on GCE node
I20260925 15:01:34.494354 19510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:34.495432 19510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:34.496651 19510 hybrid_clock.cc:648] HybridClock initialized: now 1790348494496608 us; error 34 us; skew 500 ppm
I20260925 15:01:34.497004 19510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:34 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348494, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:34.500100 19510 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:01:34.501634 19510 webserver.cc:460] Webserver started at http://127.4.243.193:33069/ using document root <none> and password file <none>
I20260925 15:01:34.502328 19510 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:34.502471 19510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:34.507853 19510 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:01:34.510576 19545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:34.512295 19510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:34.512476 19510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "ccccd14d392b43e39f8b5213d9815e65"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "4ad3e04eefb4525b916ad1df1468016e"
server_key_iv: "84c5efed8987a7d25e7c9554132161c5"
server_key_version: "encryptionkey@0"
I20260925 15:01:34.513423 19510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:34.535202 19510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:34.539515 19510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:34.539786 19510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:34.540756 19510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:34.542065 19510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:34.542203 19510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:34.542336 19510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:34.542420 19510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:34.574791 19510 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:42905
I20260925 15:01:34.574877 19657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:42905 every 8 connection(s)
I20260925 15:01:34.576213 19510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:34.577908 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 19510
I20260925 15:01:34.578553 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:32843
--local_ip_for_outbound_sockets=127.4.243.194
--tserver_master_addrs=127.4.243.254:36395
--webserver_port=38509
--webserver_interface=127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:01:34 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348494, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:34.604779 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:34.579393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:45223)
0925 15:01:34.581439 (+ 2046us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:34.581501 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:34.584236 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:34.585609 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:34.585621 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:34.586434 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:34.587285 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:34.587312 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:34.588897 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:34.588905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:34.591104 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:34.591123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:34.592188 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:34.592197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:34.592508 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:34.593264 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:34.593296 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:34.595921 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:34.600727 (+ 4806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:34.600733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:34.600744 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:34.601149 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:34.601673 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:34.601676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:34.601678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:34.601853 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:34.602889 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:34.602895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:34.603186 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:01:34.603423 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 15:01:34.603796 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"thread_start_us":1517,"threads_started":1}
I20260925 15:01:34.605468 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:34.579039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:45223 (local address 127.4.243.254:36395)
0925 15:01:34.579669 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:34.579688 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:34.581636 (+ 1948us) server_negotiation.cc:407] Connection header received
0925 15:01:34.584757 (+ 3121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:34.584770 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:34.585093 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:34.585396 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:34.587517 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:34.588735 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:34.591292 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:34.591946 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:34.592090 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:34.596229 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:34.596242 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:34.596264 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:34.596317 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:34.600588 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:34.601303 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:34.601315 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:34.601326 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:34.601552 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:34.601984 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:34.601991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:34.601996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:34.602718 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:34.602941 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:34.603711 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 15:01:34.604184 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260925 15:01:34.608275 19658 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36395
I20260925 15:01:34.608599 19658 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:34.609421 19658 heartbeater.cc:511] Master 127.4.243.254:36395 requested a full tablet report, sending...
I20260925 15:01:34.611714 19473 ts_manager.cc:194] Registered new tserver with Master: ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:34.613260 19473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:45223
W20260925 15:01:34.901553 19662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:34.901809 19662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:34.901949 19662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:34.909106 19662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:34.909224 19662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:34.909407 19662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:01:34.918747 19662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:32843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=38509
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:34.919952 19662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:34.921190 19662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:34.934315 19667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:34.934293 19668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:34.934228 19670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:34.935024 19662 server_base.cc:1034] running on GCE node
I20260925 15:01:34.935595 19662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:34.936488 19662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:34.937709 19662 hybrid_clock.cc:648] HybridClock initialized: now 1790348494937669 us; error 53 us; skew 500 ppm
I20260925 15:01:34.938043 19662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:34 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348494, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:34.941085 19662 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:01:34.942538 19662 webserver.cc:460] Webserver started at http://127.4.243.194:38509/ using document root <none> and password file <none>
I20260925 15:01:34.943251 19662 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:34.943410 19662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:34.948554 19662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:01:34.951261 19677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:34.952574 19662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:34.952746 19662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "d9af8329ab0b41209ff9163d6104fa60"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "5a0c0e164498a9a18a00e011ed7137ca"
server_key_iv: "0b3e78c14a0dc74ce1fd82d772a44985"
server_key_version: "encryptionkey@0"
I20260925 15:01:34.953691 19662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:34.968317 19662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:34.972435 19662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:34.972770 19662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:34.973788 19662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:34.975196 19662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:34.975325 19662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:34.975457 19662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:34.975548 19662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:35.007746 19662 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:32843
I20260925 15:01:35.007776 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:32843 every 8 connection(s)
I20260925 15:01:35.009243 19662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:35.013237 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 19662
I20260925 15:01:35.014003 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:34517
--local_ip_for_outbound_sockets=127.4.243.195
--tserver_master_addrs=127.4.243.254:36395
--webserver_port=45157
--webserver_interface=127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:01:35 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348494, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:35.030944 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.012041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:46873 (local address 127.4.243.254:36395)
0925 15:01:35.012220 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.012225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.013586 (+ 1361us) server_negotiation.cc:407] Connection header received
0925 15:01:35.016560 (+ 2974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.016565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.016640 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.016728 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:35.018839 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.019523 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.021014 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.021430 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.021559 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.025465 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:35.025470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:35.025473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:35.025504 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:35.028010 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.028718 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.028729 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.028731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.028857 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.029282 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.029286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.029288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.029541 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:35.029693 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.030365 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.030586 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:01:35.031875 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.012557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:46873)
0925 15:01:35.013402 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.013454 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.016180 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.017020 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.017034 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.017807 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:35.018664 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.018694 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.019681 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.019686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.020865 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.020876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.021652 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.021659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.021891 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.022581 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:35.022640 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:35.025254 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:35.028156 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:35.028164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:35.028180 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:35.028588 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:35.028967 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:35.028972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:35.028976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:35.029158 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:35.029664 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:35.029671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:35.029991 (+ 320us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.030252 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.030795 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":315,"threads_started":1}
I20260925 15:01:35.033320 19790 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36395
I20260925 15:01:35.033612 19790 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:35.034312 19790 heartbeater.cc:511] Master 127.4.243.254:36395 requested a full tablet report, sending...
I20260925 15:01:35.035948 19473 ts_manager.cc:194] Registered new tserver with Master: d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843)
I20260925 15:01:35.037204 19473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:46873
W20260925 15:01:35.348078 19793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:35.348364 19793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:35.348467 19793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:35.356137 19793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:35.356251 19793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:35.356398 19793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:01:35.365763 19793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:34517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=45157
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:35.367336 19793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:35.368810 19793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:35.381932 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:35.382046 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:35.381968 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:35.382871 19793 server_base.cc:1034] running on GCE node
I20260925 15:01:35.383489 19793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:35.384413 19793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:35.385622 19793 hybrid_clock.cc:648] HybridClock initialized: now 1790348495385538 us; error 42 us; skew 500 ppm
I20260925 15:01:35.386044 19793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:35 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348495, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:35.390451 19793 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:01:35.391973 19793 webserver.cc:460] Webserver started at http://127.4.243.195:45157/ using document root <none> and password file <none>
I20260925 15:01:35.392688 19793 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:35.392828 19793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:35.398176 19793 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:01:35.400888 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:35.402215 19793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:01:35.402388 19793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "be8eff67756749faacd8187d16dce431"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "1e1043815f9a747b40455942ff711d5a"
server_key_iv: "f6368d8112b89f7df95f5a2d561a5d47"
server_key_version: "encryptionkey@0"
I20260925 15:01:35.403429 19793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:35.420027 19793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:35.424539 19793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:35.424829 19793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:35.425812 19793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:35.427245 19793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:35.427364 19793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:35.427495 19793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:35.427584 19793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:35.461166 19793 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:34517
I20260925 15:01:35.461221 19920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:34517 every 8 connection(s)
I20260925 15:01:35.462668 19793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:35.468873 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 19793
Sep 25 15:01:35 dist-test-slave-qxh0 krb5kdc[18889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348495, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:35.484539 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.469921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.4.243.254:36395)
0925 15:01:35.470726 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.470730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.470744 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:01:35.470785 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.470788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.470835 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.470922 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:35.472671 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.473516 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.475260 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.475687 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.475820 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.478655 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:35.478660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:35.478663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:35.478703 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:35.481675 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.482445 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.482453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.482457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.482641 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.483026 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.483031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.483035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.483320 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:35.483573 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.483899 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.484121 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":571,"threads_started":1}
I20260925 15:01:35.484648 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.465305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:54183 (local address 127.4.243.254:36395)
0925 15:01:35.465492 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.465496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.466724 (+ 1228us) server_negotiation.cc:407] Connection header received
0925 15:01:35.468857 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.468862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.468938 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.469073 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:35.471042 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.471780 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.473843 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.474224 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.474313 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.478276 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:35.478282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:35.478285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:35.478322 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:35.481568 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.482298 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.482303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.482307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.482489 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.482957 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.482962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.482966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.483237 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:35.483400 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.484091 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.484300 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:01:35.485975 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.465806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:54183)
0925 15:01:35.466528 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.466565 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.468538 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.469340 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.469351 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.469989 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:35.470839 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.470866 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.471956 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.471962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.473693 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.473710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.474442 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.474450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.474753 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.475554 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:35.475576 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:35.478046 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:35.481707 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:35.481716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:35.481734 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:35.482153 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:35.482628 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:35.482633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:35.482637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:35.482833 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:35.483373 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:35.483381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:35.483721 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.483991 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.484526 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":269,"threads_started":1}
I20260925 15:01:35.487821 19921 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36395
I20260925 15:01:35.488168 19921 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:35.488906 19921 heartbeater.cc:511] Master 127.4.243.254:36395 requested a full tablet report, sending...
I20260925 15:01:35.490301 19473 ts_manager.cc:194] Registered new tserver with Master: be8eff67756749faacd8187d16dce431 (127.4.243.195:34517)
I20260925 15:01:35.491185 19473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:54183
I20260925 15:01:35.499676 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:35.506299 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.500524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.4.243.254:36395)
0925 15:01:35.500797 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.500844 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.501205 (+ 361us) server_negotiation.cc:407] Connection header received
0925 15:01:35.501384 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.501388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.501521 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.501642 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:35.502035 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.502777 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.504267 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.504669 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.504768 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.505232 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:35.505692 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:35.505834 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.505939 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.506004 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 15:01:35.507237 19473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:35.514358 19473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:35.538386 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.525212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:34517 (local address 127.0.0.1:60420)
0925 15:01:35.526111 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.526164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.526394 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.527553 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.527564 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.527603 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:35.528056 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.528071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.530974 (+ 2903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.530983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.534345 (+ 3362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.534367 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.537050 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.537059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.537355 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.537429 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.537610 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.537865 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":839,"threads_started":1}
I20260925 15:01:35.539415 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.525350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.4.243.195:34517)
0925 15:01:35.525998 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.526006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.526628 (+ 622us) server_negotiation.cc:407] Connection header received
0925 15:01:35.526754 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.526766 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.527035 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.527227 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:35.528310 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.530782 (+ 2472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.534574 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.536468 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.536618 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.536896 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.538731 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.538969 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":318,"threads_started":1}
I20260925 15:01:35.543752 19856 tablet_service.cc:1467] Processing CreateTablet for tablet 63370e7d99c84519996780a97e6afc6f (DEFAULT_TABLE table=test-table [id=4764688beb0749c5aa81ced6774e3a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:35.545164 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63370e7d99c84519996780a97e6afc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:35.547245 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.525779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48906)
0925 15:01:35.536323 (+ 10544us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.536349 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.536506 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.539683 (+ 3177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.539688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.539715 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:35.540110 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.540122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.541975 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.541979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.544020 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.544034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.546285 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.546291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.546488 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.546521 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.546674 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.546831 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10356,"mutex_wait_us":32,"thread_start_us":10996,"threads_started":1}
I20260925 15:01:35.547446 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.526344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.4.243.193:42905)
0925 15:01:35.538972 (+ 12628us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.538981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.539016 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:01:35.539095 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.539102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.539300 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.539517 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:35.540304 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.541814 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.544183 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.546050 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.546203 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.546390 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.546911 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.547092 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12482,"thread_start_us":12396,"threads_started":1}
I20260925 15:01:35.547832 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.527295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.4.243.194:32843)
0925 15:01:35.540584 (+ 13289us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.540590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.540614 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:01:35.540749 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.540756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.540970 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.541160 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:35.541725 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.543051 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.545024 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.546430 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.546598 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.546807 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.547339 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.547512 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13141,"thread_start_us":13040,"threads_started":1}
I20260925 15:01:35.548251 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.526983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:32843 (local address 127.0.0.1:60472)
0925 15:01:35.537125 (+ 10142us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.537145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.537260 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.541145 (+ 3885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.541149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.541175 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:35.541505 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.541520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.543221 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.543224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.544850 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.544866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.546687 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.546694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.546907 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.546942 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.547030 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.547143 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9984,"mutex_wait_us":25,"thread_start_us":10355,"threads_started":1}
I20260925 15:01:35.551410 19593 tablet_service.cc:1467] Processing CreateTablet for tablet 63370e7d99c84519996780a97e6afc6f (DEFAULT_TABLE table=test-table [id=4764688beb0749c5aa81ced6774e3a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:35.551636 19725 tablet_service.cc:1467] Processing CreateTablet for tablet 63370e7d99c84519996780a97e6afc6f (DEFAULT_TABLE table=test-table [id=4764688beb0749c5aa81ced6774e3a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:35.552670 19725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63370e7d99c84519996780a97e6afc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:35.552722 19593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63370e7d99c84519996780a97e6afc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:35.562001 19937 tablet_bootstrap.cc:492] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Bootstrap starting.
I20260925 15:01:35.564873 19938 tablet_bootstrap.cc:492] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Bootstrap starting.
I20260925 15:01:35.565881 19937 tablet_bootstrap.cc:654] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:35.567361 19937 log.cc:824] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:35.567404 19939 tablet_bootstrap.cc:492] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Bootstrap starting.
I20260925 15:01:35.568531 19938 tablet_bootstrap.cc:654] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:35.569458 19938 log.cc:824] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:35.571265 19939 tablet_bootstrap.cc:654] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:35.572276 19938 tablet_bootstrap.cc:492] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: No bootstrap required, opened a new log
I20260925 15:01:35.572578 19938 ts_tablet_manager.cc:1397] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:01:35.572628 19939 log.cc:824] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:35.573439 19937 tablet_bootstrap.cc:492] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: No bootstrap required, opened a new log
I20260925 15:01:35.573807 19937 ts_tablet_manager.cc:1397] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:01:35.582821 19938 raft_consensus.cc:348] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.583220 19938 raft_consensus.cc:374] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:35.583300 19938 raft_consensus.cc:729] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9af8329ab0b41209ff9163d6104fa60, State: Initialized, Role: FOLLOWER
I20260925 15:01:35.583880 19938 consensus_queue.cc:260] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.585258 19939 tablet_bootstrap.cc:492] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: No bootstrap required, opened a new log
I20260925 15:01:35.585532 19939 ts_tablet_manager.cc:1397] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260925 15:01:35.587404 19937 raft_consensus.cc:348] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.587865 19937 raft_consensus.cc:374] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:35.587993 19937 raft_consensus.cc:729] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8eff67756749faacd8187d16dce431, State: Initialized, Role: FOLLOWER
I20260925 15:01:35.588578 19937 consensus_queue.cc:260] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.591755 19938 ts_tablet_manager.cc:1428] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Time spent starting tablet: real 0.019s user 0.009s sys 0.010s
I20260925 15:01:35.592096 19790 heartbeater.cc:503] Master 127.4.243.254:36395 was elected leader, sending a full tablet report...
I20260925 15:01:35.593931 19921 heartbeater.cc:503] Master 127.4.243.254:36395 was elected leader, sending a full tablet report...
I20260925 15:01:35.594641 19937 ts_tablet_manager.cc:1428] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260925 15:01:35.597508 19939 raft_consensus.cc:348] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.597957 19939 raft_consensus.cc:374] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:35.598073 19939 raft_consensus.cc:729] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccccd14d392b43e39f8b5213d9815e65, State: Initialized, Role: FOLLOWER
I20260925 15:01:35.598676 19939 consensus_queue.cc:260] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.599572 19658 heartbeater.cc:503] Master 127.4.243.254:36395 was elected leader, sending a full tablet report...
I20260925 15:01:35.600169 19939 ts_tablet_manager.cc:1428] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 15:01:35.657990 19943 raft_consensus.cc:482] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:35.658115 19944 raft_consensus.cc:482] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:35.658365 19943 raft_consensus.cc:504] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.658362 19944 raft_consensus.cc:504] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.660511 19943 leader_election.cc:288] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be8eff67756749faacd8187d16dce431 (127.4.243.195:34517), ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:35.668285 19944 leader_election.cc:288] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843), ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:35.681229 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.671803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:58959 (local address 127.4.243.194:32843)
0925 15:01:35.672019 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.672027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.672049 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:01:35.672145 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.672153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.672267 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.672380 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:35.673433 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.675805 (+ 2372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.678332 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.679761 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.679919 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.680002 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.680821 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.680958 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":53}
I20260925 15:01:35.681479 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.661354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:48931 (local address 127.4.243.195:34517)
0925 15:01:35.661547 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.661552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.671474 (+ 9922us) server_negotiation.cc:407] Connection header received
0925 15:01:35.673231 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.673239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.673323 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.673394 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:35.675416 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.676766 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.678896 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.680324 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.680460 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.680538 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.681074 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.681199 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":59}
I20260925 15:01:35.681576 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.660732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:32843 (local address 127.4.243.195:58959)
0925 15:01:35.661126 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.661148 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.661289 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.672600 (+ 11311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.672608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.672654 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:35.673198 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.673223 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.675973 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.675981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.678138 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.678152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.680088 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.680094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.680335 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.680370 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.680459 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.680597 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1}
I20260925 15:01:35.682344 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.660732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:34517 (local address 127.4.243.194:48931)
0925 15:01:35.671343 (+ 10611us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.671369 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.673044 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.673603 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.673613 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.673641 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:35.675248 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.675267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.676905 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.676910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.678686 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.678700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.680680 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.680686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.680888 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.680925 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.681645 (+ 720us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.681851 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10434,"mutex_wait_us":45,"thread_start_us":10757,"threads_started":1}
I20260925 15:01:35.682723 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.661666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:48347 (local address 127.4.243.193:42905)
0925 15:01:35.661914 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.661919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.675405 (+ 13486us) server_negotiation.cc:407] Connection header received
0925 15:01:35.675668 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.675673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.675746 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.675820 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:35.676545 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.677665 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.680194 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.681446 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.681571 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.681639 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.682258 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.682412 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":71}
I20260925 15:01:35.682626 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9af8329ab0b41209ff9163d6104fa60" is_pre_election: true
I20260925 15:01:35.683128 19745 raft_consensus.cc:2454] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8eff67756749faacd8187d16dce431 in term 0.
I20260925 15:01:35.683630 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.661126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.4.243.194:48347)
0925 15:01:35.675278 (+ 14152us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.675299 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.675505 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.675995 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.676000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.676023 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:35.676369 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.676380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.677823 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.677829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.680018 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.680033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.681841 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.681846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.682045 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.682078 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.682920 (+ 842us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.683053 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14020,"mutex_wait_us":1,"thread_start_us":13804,"threads_started":1}
I20260925 15:01:35.684558 19613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "d9af8329ab0b41209ff9163d6104fa60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccccd14d392b43e39f8b5213d9815e65" is_pre_election: true
I20260925 15:01:35.684047 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "d9af8329ab0b41209ff9163d6104fa60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8eff67756749faacd8187d16dce431" is_pre_election: true
I20260925 15:01:35.685025 19812 leader_election.cc:302] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8eff67756749faacd8187d16dce431, d9af8329ab0b41209ff9163d6104fa60; no voters:
I20260925 15:01:35.685102 19613 raft_consensus.cc:2454] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9af8329ab0b41209ff9163d6104fa60 in term 0.
I20260925 15:01:35.685338 19876 raft_consensus.cc:2454] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9af8329ab0b41209ff9163d6104fa60 in term 0.
I20260925 15:01:35.685537 19944 raft_consensus.cc:2781] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:35.685695 19944 raft_consensus.cc:482] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:35.685896 19944 raft_consensus.cc:3037] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:35.685811 19679 leader_election.cc:302] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccccd14d392b43e39f8b5213d9815e65, d9af8329ab0b41209ff9163d6104fa60; no voters:
I20260925 15:01:35.686265 19943 raft_consensus.cc:2781] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:35.686416 19943 raft_consensus.cc:482] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:35.686512 19943 raft_consensus.cc:3037] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:35.686569 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.660916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.4.243.195:33497)
0925 15:01:35.674120 (+ 13204us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.674138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:35.674246 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:35.674670 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:35.674674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:35.675393 (+ 719us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:35.675888 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.675902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.678690 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.678695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.680869 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:35.680881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.684060 (+ 3179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.684069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:35.685075 (+ 1006us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.685112 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:01:35.685195 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:01:35.685310 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13033,"mutex_wait_us":65,"thread_start_us":12831,"threads_started":1}
I20260925 15:01:35.686770 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.661784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:33497 (local address 127.4.243.193:42905)
0925 15:01:35.674227 (+ 12443us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.674233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.674255 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:01:35.674421 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.674428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.674508 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.674686 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:35.676137 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.678507 (+ 2370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.681049 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.683719 (+ 2670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.683881 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.683955 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.685420 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.685547 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":12302,"thread_start_us":12305,"threads_started":1}
I20260925 15:01:35.687402 19613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccccd14d392b43e39f8b5213d9815e65" is_pre_election: true
I20260925 15:01:35.687623 19613 raft_consensus.cc:2454] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8eff67756749faacd8187d16dce431 in term 0.
I20260925 15:01:35.688961 19944 raft_consensus.cc:504] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.689046 19943 raft_consensus.cc:504] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.689628 19944 leader_election.cc:288] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 election: Requested vote from peers d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843), ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:35.689738 19943 leader_election.cc:288] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [CANDIDATE]: Term 1 election: Requested vote from peers be8eff67756749faacd8187d16dce431 (127.4.243.195:34517), ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905)
I20260925 15:01:35.690114 19745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9af8329ab0b41209ff9163d6104fa60"
I20260925 15:01:35.690145 19613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "d9af8329ab0b41209ff9163d6104fa60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccccd14d392b43e39f8b5213d9815e65"
I20260925 15:01:35.690137 19876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "d9af8329ab0b41209ff9163d6104fa60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8eff67756749faacd8187d16dce431"
I20260925 15:01:35.690321 19613 raft_consensus.cc:3037] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:35.690397 19745 raft_consensus.cc:2379] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate be8eff67756749faacd8187d16dce431 in current term 1: Already voted for candidate d9af8329ab0b41209ff9163d6104fa60 in this term.
I20260925 15:01:35.690403 19876 raft_consensus.cc:2379] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d9af8329ab0b41209ff9163d6104fa60 in current term 1: Already voted for candidate be8eff67756749faacd8187d16dce431 in this term.
I20260925 15:01:35.691502 19612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63370e7d99c84519996780a97e6afc6f" candidate_uuid: "be8eff67756749faacd8187d16dce431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccccd14d392b43e39f8b5213d9815e65"
I20260925 15:01:35.692322 19810 leader_election.cc:302] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be8eff67756749faacd8187d16dce431; no voters: ccccd14d392b43e39f8b5213d9815e65, d9af8329ab0b41209ff9163d6104fa60
I20260925 15:01:35.692615 19944 raft_consensus.cc:2726] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 15:01:35.693416 19613 raft_consensus.cc:2454] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9af8329ab0b41209ff9163d6104fa60 in term 1.
I20260925 15:01:35.693755 19679 leader_election.cc:302] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ccccd14d392b43e39f8b5213d9815e65, d9af8329ab0b41209ff9163d6104fa60; no voters: be8eff67756749faacd8187d16dce431
I20260925 15:01:35.694101 19943 raft_consensus.cc:2781] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:35.694585 19943 raft_consensus.cc:686] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 1 LEADER]: Becoming Leader. State: Replica: d9af8329ab0b41209ff9163d6104fa60, State: Running, Role: LEADER
I20260925 15:01:35.695418 19943 consensus_queue.cc:237] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } }
I20260925 15:01:35.699621 19472 catalog_manager.cc:5495] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194). New cstate: current_term: 1 leader_uuid: "d9af8329ab0b41209ff9163d6104fa60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9af8329ab0b41209ff9163d6104fa60" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 } health_report { overall_health: UNKNOWN } } }
W20260925 15:01:35.714295 19922 tablet.cc:2369] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:35.743836 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.738978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.4.243.194:32843)
0925 15:01:35.739136 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.739140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.739156 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:01:35.739337 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.739341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.739413 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.739535 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:35.739849 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.740562 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.741850 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.742220 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.742340 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.742842 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:35.743325 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:35.743515 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.743554 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.743617 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:01:35.751539 19613 raft_consensus.cc:1261] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Refusing update from remote peer d9af8329ab0b41209ff9163d6104fa60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:35.751559 19876 raft_consensus.cc:1261] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Refusing update from remote peer d9af8329ab0b41209ff9163d6104fa60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:35.752358 19950 consensus_queue.cc:1035] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be8eff67756749faacd8187d16dce431" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:35.752686 19943 consensus_queue.cc:1035] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccccd14d392b43e39f8b5213d9815e65" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:01:35.761327 19791 tablet.cc:2369] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:35.762976 19953 mvcc.cc:205] Tried to move back new op lower bound from 7333267438589190144 to 7333267438372081664. Current Snapshot: MvccSnapshot[applied={T|T < 7333267438589190144}]
I20260925 15:01:35.764026 19954 mvcc.cc:205] Tried to move back new op lower bound from 7333267438589190144 to 7333267438372081664. Current Snapshot: MvccSnapshot[applied={T|T < 7333267438589190144}]
I20260925 15:01:35.767539 19955 mvcc.cc:205] Tried to move back new op lower bound from 7333267438589190144 to 7333267438372081664. Current Snapshot: MvccSnapshot[applied={T|T < 7333267438589190144}]
I20260925 15:01:35.781494 19472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:35.781739 19472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:35.785199 19472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Sending DeleteTablet for 3 replicas of tablet 63370e7d99c84519996780a97e6afc6f
I20260925 15:01:35.786350 19856 tablet_service.cc:1514] Processing DeleteTablet for tablet 63370e7d99c84519996780a97e6afc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:35 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:60420
I20260925 15:01:35.786631 19593 tablet_service.cc:1514] Processing DeleteTablet for tablet 63370e7d99c84519996780a97e6afc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:35 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:48906
I20260925 15:01:35.787086 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 19510
I20260925 15:01:35.787302 19725 tablet_service.cc:1514] Processing DeleteTablet for tablet 63370e7d99c84519996780a97e6afc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:35 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:60472
I20260925 15:01:35.788125 19964 tablet_replica.cc:330] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: stopping tablet replica
I20260925 15:01:35.788677 19964 raft_consensus.cc:2229] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:35.789273 19964 raft_consensus.cc:2258] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:35.790155 19966 tablet_replica.cc:330] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: stopping tablet replica
I20260925 15:01:35.790575 19966 raft_consensus.cc:2229] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:35.791283 19966 raft_consensus.cc:2258] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:35.791599 19964 ts_tablet_manager.cc:1905] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:35.793192 19966 ts_tablet_manager.cc:1905] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:35.798412 19964 ts_tablet_manager.cc:1918] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:35.798565 19964 log.cc:1196] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/63370e7d99c84519996780a97e6afc6f
I20260925 15:01:35.799162 19964 ts_tablet_manager.cc:1939] T 63370e7d99c84519996780a97e6afc6f P be8eff67756749faacd8187d16dce431: Deleting consensus metadata
I20260925 15:01:35.799162 19965 tablet_replica.cc:330] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: stopping tablet replica
I20260925 15:01:35.799671 19965 raft_consensus.cc:2229] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:35.800201 19965 raft_consensus.cc:2258] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:35.800954 19460 catalog_manager.cc:4841] TS be8eff67756749faacd8187d16dce431 (127.4.243.195:34517): tablet 63370e7d99c84519996780a97e6afc6f (table test-table [id=4764688beb0749c5aa81ced6774e3a61]) successfully deleted
I20260925 15:01:35.802428 19966 ts_tablet_manager.cc:1918] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:35.802680 19966 log.cc:1196] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/63370e7d99c84519996780a97e6afc6f
I20260925 15:01:35.803092 19965 ts_tablet_manager.cc:1905] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:35.803396 19966 ts_tablet_manager.cc:1939] T 63370e7d99c84519996780a97e6afc6f P d9af8329ab0b41209ff9163d6104fa60: Deleting consensus metadata
I20260925 15:01:35.805027 19460 catalog_manager.cc:4841] TS d9af8329ab0b41209ff9163d6104fa60 (127.4.243.194:32843): tablet 63370e7d99c84519996780a97e6afc6f (table test-table [id=4764688beb0749c5aa81ced6774e3a61]) successfully deleted
I20260925 15:01:35.806694 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.787846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.4.243.193:42905)
0925 15:01:35.788007 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.788013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.788123 (+ 110us) server_negotiation.cc:407] Connection header received
0925 15:01:35.788332 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.788340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.788443 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.788540 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:35.790319 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.791965 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.793968 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.794856 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.794983 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.798728 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:35.798747 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:35.798765 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:35.798821 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:35.803040 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.804075 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.804083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.804087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.804278 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.804771 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.804777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.804786 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.805287 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:35.805583 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.806032 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.806286 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:01:35.807425 19653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:35.929900 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 19510
W20260925 15:01:35.947773 19458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:42905 (error 108)
I20260925 15:01:35.947968 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 19662
I20260925 15:01:35.949790 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.949295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48924)
0925 15:01:35.949477 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:35.949613 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 15:01:35.950083 19458 catalog_manager.cc:4584] TS ccccd14d392b43e39f8b5213d9815e65 (127.4.243.193:42905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63370e7d99c84519996780a97e6afc6f: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
I20260925 15:01:35.964125 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:35.949010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.4.243.194:32843)
0925 15:01:35.949206 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:35.949213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:35.949232 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:01:35.949348 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:35.949356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:35.949447 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:35.949583 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:35.951441 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.952187 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.954041 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:35.954444 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:35.954534 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:35.957990 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:35.958003 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:35.958018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:35.958047 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:35.961108 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.962003 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.962011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.962013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.962161 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:35.962546 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:35.962549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:35.962552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:35.963113 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:35.963339 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:35.963667 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:01:35.963833 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:01:35.964890 19785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:36.003746 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.003184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48936)
0925 15:01:36.003426 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:36.003546 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 15:01:36.004868 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.004461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48952)
0925 15:01:36.004622 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:36.004725 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:01:36.065610 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.065071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48966)
0925 15:01:36.065308 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:36.065421 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 15:01:36.066715 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.066327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48970)
0925 15:01:36.066477 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:36.066556 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:01:36.083586 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 19662
I20260925 15:01:36.109000 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 19793
I20260925 15:01:36.123860 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.109773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.4.243.195:34517)
0925 15:01:36.109958 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:36.109963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:36.110123 (+ 160us) server_negotiation.cc:407] Connection header received
0925 15:01:36.110319 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:36.110323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:36.110399 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:36.110484 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:36.112197 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:36.112913 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:36.114874 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:36.115257 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:36.115371 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:36.118444 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:36.118457 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:36.118474 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:36.118504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:36.121237 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:36.121870 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:36.121877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:36.121879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:36.122030 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:36.122358 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:36.122361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:36.122363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:36.122818 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:36.123035 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:36.123275 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:01:36.123438 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:01:36.124508 19916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:36.147338 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.146772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48980)
0925 15:01:36.147014 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:36.147121 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 15:01:36.148777 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.148222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:48994)
0925 15:01:36.148528 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:36.148626 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 15:01:36.220084 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 19793
I20260925 15:01:36.236677 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 19440
I20260925 15:01:36.237778 19503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:36.349449 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:36.348816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42905 (local address 127.0.0.1:49010)
0925 15:01:36.349020 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:36.349148 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:01:36.349512 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 19440
I20260925 15:01:36.367479 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36395
--webserver_interface=127.4.243.254
--webserver_port=45161
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:36.690749 19972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:36.691083 19972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:36.691242 19972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:36.698318 19972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:36.698415 19972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:36.698532 19972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:36.698637 19972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:36.698701 19972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:36.708313 19972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=45161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:36.709690 19972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:36.711043 19972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:36.722191 19977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:36.722401 19978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:36.722504 19980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:36.723279 19972 server_base.cc:1034] running on GCE node
I20260925 15:01:36.724009 19972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:36.725349 19972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:36.726572 19972 hybrid_clock.cc:648] HybridClock initialized: now 1790348496726525 us; error 68 us; skew 500 ppm
I20260925 15:01:36.726943 19972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:36 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348496, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:36.736148 19972 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:36.737876 19972 webserver.cc:460] Webserver started at http://127.4.243.254:45161/ using document root <none> and password file <none>
I20260925 15:01:36.738576 19972 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:36.738765 19972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:36.744001 19972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:01:36.746745 19987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:36.747988 19972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:01:36.748157 19972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "2d297576003941aa992d9e6e9629baec"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "1ed4b73014951749794d2ae41f11421e"
server_key_iv: "1961be7e9374950e015229cf0b8c7bd3"
server_key_version: "encryptionkey@0"
I20260925 15:01:36.749082 19972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:36.791357 19972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:36.795392 19972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:36.795660 19972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:36.814476 19972 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:36395
I20260925 15:01:36.814482 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:36395 every 8 connection(s)
I20260925 15:01:36.815968 19972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:36.819787 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 19972
I20260925 15:01:36.820767 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:42905
--local_ip_for_outbound_sockets=127.4.243.193
--tserver_master_addrs=127.4.243.254:36395
--webserver_port=33069
--webserver_interface=127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:36.824149 20039 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:01:36.829212 20039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Bootstrap starting.
I20260925 15:01:36.849061 20039 log.cc:824] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:36.865226 20039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:01:36.865809 20039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec: Bootstrap complete.
I20260925 15:01:36.875101 20039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:36.875910 20039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d297576003941aa992d9e6e9629baec, State: Initialized, Role: FOLLOWER
I20260925 15:01:36.876497 20039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:36.876709 20039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:36.876844 20039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:36.877014 20039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:01:36.880816 20039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:36.881278 20039 leader_election.cc:302] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d297576003941aa992d9e6e9629baec; no voters:
I20260925 15:01:36.881942 20039 leader_election.cc:288] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:01:36.882093 20043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:01:36.883145 20043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [term 3 LEADER]: Becoming Leader. State: Replica: 2d297576003941aa992d9e6e9629baec, State: Running, Role: LEADER
I20260925 15:01:36.883682 20043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } }
I20260925 15:01:36.884060 20039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:36.886248 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2d297576003941aa992d9e6e9629baec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } } }
I20260925 15:01:36.886677 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:36.891491 20045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d297576003941aa992d9e6e9629baec. Latest consensus state: current_term: 3 leader_uuid: "2d297576003941aa992d9e6e9629baec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d297576003941aa992d9e6e9629baec" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36395 } } }
I20260925 15:01:36.891882 20045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d297576003941aa992d9e6e9629baec [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:36.898939 20049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:36.907577 20049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:36.909021 20049 catalog_manager.cc:1259] Loaded cluster ID: 5b48b18132e04a24a75ebfc1c7491a35
I20260925 15:01:36.909164 20049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:01:36.911152 20049 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2bda1879ad google::LogMessage::Fail() at ??:0
@ 0x7f2bda18a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2bda1874de google::LogMessage::Flush() at ??:0
@ 0x7f2bda188a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2befbba512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2befbb7976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2befc3f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2befc3f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2bdcdd3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2bdcdd691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2bdcdd6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2bdcdb1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2be008b6db start_thread at ??:0
@ 0x7f2bd902161f clone at ??:0
W20260925 15:01:37.174214 20041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:37.174556 20041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:37.174774 20041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:37.182096 20041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:37.182199 20041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:37.182488 20041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:01:37.193084 20041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:42905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=33069
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:37.194507 20041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:37.195904 20041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:37.209100 20080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.209067 20083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.209158 20081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:37.210165 20041 server_base.cc:1034] running on GCE node
I20260925 15:01:37.210839 20041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:37.211750 20041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:37.212960 20041 hybrid_clock.cc:648] HybridClock initialized: now 1790348497212900 us; error 61 us; skew 500 ppm
I20260925 15:01:37.213246 20041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:37 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:37.216355 20041 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:01:37.217710 20041 webserver.cc:460] Webserver started at http://127.4.243.193:33069/ using document root <none> and password file <none>
I20260925 15:01:37.218379 20041 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:37.218474 20041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:37.223608 20041 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:01:37.226405 20090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.227739 20041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:01:37.227916 20041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "ccccd14d392b43e39f8b5213d9815e65"
format_stamp: "Formatted at 2026-09-25 15:01:31 on dist-test-slave-qxh0"
server_key: "4ad3e04eefb4525b916ad1df1468016e"
server_key_iv: "84c5efed8987a7d25e7c9554132161c5"
server_key_version: "encryptionkey@0"
I20260925 15:01:37.228825 20041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:37.261811 20041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:37.266330 20041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:37.266624 20041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:37.267628 20041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:37.269352 20097 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 15:01:37.274787 20097 ts_tablet_manager.cc:1852] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 15:01:37.275759 20097 ts_tablet_manager.cc:1905] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:37.282543 20097 ts_tablet_manager.cc:1918] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:37.282778 20097 log.cc:1196] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/63370e7d99c84519996780a97e6afc6f
I20260925 15:01:37.283229 20097 ts_tablet_manager.cc:1939] T 63370e7d99c84519996780a97e6afc6f P ccccd14d392b43e39f8b5213d9815e65: Deleting consensus metadata
I20260925 15:01:37.283623 20041 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 15:01:37.283736 20041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260925 15:01:37.283897 20041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:37.283982 20041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.317420 20041 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:42905
I20260925 15:01:37.317466 20203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:42905 every 8 connection(s)
I20260925 15:01:37.318848 20041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:37.323647 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.321809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:57399)
0925 15:01:37.322528 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.322833 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":266,"threads_started":1}
W20260925 15:01:37.324256 20204 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:36395 (0 consecutive failures): Network error: Failed to ping master at 127.4.243.254:36395: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
I20260925 15:01:37.325136 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.324799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:56865)
0925 15:01:37.324962 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.325037 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:01:37.326162 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.325697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:50985)
0925 15:01:37.325899 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.326003 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:01:37.326231 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 20041
W20260925 15:01:37.326457 20204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:01:37.326975 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:32843
--local_ip_for_outbound_sockets=127.4.243.194
--tserver_master_addrs=127.4.243.254:36395
--webserver_port=38509
--webserver_interface=127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:37.654186 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:37.654458 20207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:37.654554 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:37.661808 20207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:37.661908 20207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:37.662025 20207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:01:37.671525 20207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:32843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=38509
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:37.672689 20207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:37.674144 20207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:37.686926 20212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.686903 20213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.687227 20215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:37.692203 20207 server_base.cc:1034] running on GCE node
I20260925 15:01:37.692896 20207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:37.693878 20207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:37.695091 20207 hybrid_clock.cc:648] HybridClock initialized: now 1790348497695039 us; error 36 us; skew 500 ppm
I20260925 15:01:37.695391 20207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:37 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:37.698527 20207 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:01:37.700042 20207 webserver.cc:460] Webserver started at http://127.4.243.194:38509/ using document root <none> and password file <none>
I20260925 15:01:37.700719 20207 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:37.700819 20207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:37.706022 20207 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:01:37.708645 20222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.709993 20207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:01:37.710121 20207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "d9af8329ab0b41209ff9163d6104fa60"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "5a0c0e164498a9a18a00e011ed7137ca"
server_key_iv: "0b3e78c14a0dc74ce1fd82d772a44985"
server_key_version: "encryptionkey@0"
I20260925 15:01:37.711098 20207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:37.734411 20207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:37.738376 20207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:37.738731 20207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:37.739684 20207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:37.741015 20207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:37.741108 20207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.741254 20207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:37.741335 20207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.773991 20207 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:32843
I20260925 15:01:37.774078 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:32843 every 8 connection(s)
I20260925 15:01:37.775533 20207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:37.779529 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 20207
I20260925 15:01:37.780297 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:34517
--local_ip_for_outbound_sockets=127.4.243.195
--tserver_master_addrs=127.4.243.254:36395
--webserver_port=45157
--webserver_interface=127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:37.780655 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.778371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:55919)
0925 15:01:37.779158 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.779493 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":324,"threads_started":1}
W20260925 15:01:37.781376 20335 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:36395 (0 consecutive failures): Network error: Failed to ping master at 127.4.243.254:36395: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
I20260925 15:01:37.782274 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.781864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:33797)
0925 15:01:37.782041 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.782129 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:01:37.783222 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.782813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:37843)
0925 15:01:37.782978 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.783074 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:01:37.783445 20335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:01:38.102598 20338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:38.102931 20338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:38.103025 20338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:38.109913 20338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:38.110005 20338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:38.110110 20338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:01:38.119459 20338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:34517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=45157
--tserver_master_addrs=127.4.243.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:38.120607 20338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:38.121872 20338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:38.135270 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:38.135324 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:38.135234 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:38.141426 20338 server_base.cc:1034] running on GCE node
I20260925 15:01:38.142042 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:38.143108 20338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:38.144340 20338 hybrid_clock.cc:648] HybridClock initialized: now 1790348498144290 us; error 41 us; skew 500 ppm
I20260925 15:01:38.144659 20338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:38 dist-test-slave-qxh0 krb5kdc[18889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348498, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:38.147728 20338 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:01:38.149247 20338 webserver.cc:460] Webserver started at http://127.4.243.195:45157/ using document root <none> and password file <none>
I20260925 15:01:38.149950 20338 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:38.150166 20338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:38.155277 20338 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:01:38.157760 20353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:38.159168 20338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:01:38.159344 20338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "be8eff67756749faacd8187d16dce431"
format_stamp: "Formatted at 2026-09-25 15:01:32 on dist-test-slave-qxh0"
server_key: "1e1043815f9a747b40455942ff711d5a"
server_key_iv: "f6368d8112b89f7df95f5a2d561a5d47"
server_key_version: "encryptionkey@0"
I20260925 15:01:38.160252 20338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:38.176609 20338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:38.180670 20338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:38.180958 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:38.181928 20338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:38.183326 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:38.183450 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:38.183573 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:38.183674 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:38.216958 20338 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:34517
I20260925 15:01:38.217008 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:34517 every 8 connection(s)
I20260925 15:01:38.218406 20338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:38.222473 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 20338
I20260925 15:01:38.223412 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.221442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:47709)
0925 15:01:38.222131 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.222345 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":277,"threads_started":1}
W20260925 15:01:38.224069 20466 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:36395 (0 consecutive failures): Network error: Failed to ping master at 127.4.243.254:36395: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
I20260925 15:01:38.224932 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.224590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:54201)
0925 15:01:38.224752 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.224831 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:01:38.225669 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.225400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:41239)
0925 15:01:38.225510 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.225576 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:01:38.226025 20466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:01:38.327736 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.327021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:34911)
0925 15:01:38.327430 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.327533 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":200,"threads_started":1}
I20260925 15:01:38.784646 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.783931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:56599)
0925 15:01:38.784336 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.784422 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":191,"threads_started":1}
I20260925 15:01:39.227512 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:39.226549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:56077)
0925 15:01:39.227108 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:39.227230 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":229,"threads_started":1}
I20260925 15:01:39.329449 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:39.328716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:55617)
0925 15:01:39.329198 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:39.329282 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":272,"threads_started":1}
I20260925 15:01:39.786264 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:39.785525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:47243)
0925 15:01:39.786026 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:39.786105 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":232,"threads_started":1}
I20260925 15:01:40.229013 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:40.228358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:47147)
0925 15:01:40.228792 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:40.228866 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260925 15:01:40.331081 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:40.330364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:36725)
0925 15:01:40.330833 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:40.330914 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260925 15:01:40.787878 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:40.787180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:57215)
0925 15:01:40.787605 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:40.787685 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":229,"threads_started":1}
I20260925 15:01:41.230516 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:41.229863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:36739)
0925 15:01:41.230279 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:41.230358 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260925 15:01:41.332651 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:41.331920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:60125)
0925 15:01:41.332412 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:41.332492 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":217,"threads_started":1}
I20260925 15:01:41.789428 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:41.788728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:52093)
0925 15:01:41.789175 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:41.789253 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
I20260925 15:01:42.232141 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:42.231434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:50119)
0925 15:01:42.231833 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:42.231918 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":194,"threads_started":1}
I20260925 15:01:42.334233 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:42.333536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:60251)
0925 15:01:42.333992 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:42.334073 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":230,"threads_started":1}
I20260925 15:01:42.791028 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:42.790270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:54397)
0925 15:01:42.790789 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:42.790869 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":238,"threads_started":1}
I20260925 15:01:43.233677 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:43.233033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:35383)
0925 15:01:43.233455 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:43.233528 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":214,"threads_started":1}
I20260925 15:01:43.335814 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:43.335121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:53377)
0925 15:01:43.335568 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:43.335651 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260925 15:01:43.792542 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:43.791874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:43227)
0925 15:01:43.792303 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:43.792391 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":203,"threads_started":1}
I20260925 15:01:44.235244 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:44.234481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:47659)
0925 15:01:44.234990 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:44.235092 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260925 15:01:44.337553 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:44.336725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:35569)
0925 15:01:44.337266 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:44.337353 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":249,"threads_started":1}
I20260925 15:01:44.794283 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:44.793515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:49861)
0925 15:01:44.793979 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:44.794061 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":196,"threads_started":1}
I20260925 15:01:45.236797 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:45.236039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:59545)
0925 15:01:45.236516 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:45.236595 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":212,"threads_started":1}
I20260925 15:01:45.339071 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:45.338432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:33525)
0925 15:01:45.338834 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:45.338911 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":224,"threads_started":1}
I20260925 15:01:45.795732 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:45.795142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:42757)
0925 15:01:45.795516 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:45.795588 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":193,"threads_started":1}
I20260925 15:01:46.238346 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:46.237721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:50627)
0925 15:01:46.238115 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:46.238186 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":202,"threads_started":1}
I20260925 15:01:46.340689 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:46.340003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:56709)
0925 15:01:46.340433 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:46.340528 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":208,"threads_started":1}
I20260925 15:01:46.797297 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:46.796606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:50881)
0925 15:01:46.797055 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:46.797130 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":230,"threads_started":1}
I20260925 15:01:47.239934 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:47.239220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:41225)
0925 15:01:47.239679 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:47.239773 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":223,"threads_started":1}
I20260925 15:01:47.342274 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:47.341589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:36359)
0925 15:01:47.342027 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:47.342122 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":258,"threads_started":1}
I20260925 15:01:47.799127 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:47.798301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:36383)
0925 15:01:47.798887 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:47.798971 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
I20260925 15:01:48.241346 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:48.240738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:46639)
0925 15:01:48.241126 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:48.241196 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":215,"threads_started":1}
I20260925 15:01:48.344017 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:48.343293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:43427)
0925 15:01:48.343743 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:48.343826 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260925 15:01:48.800841 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:48.800006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:40541)
0925 15:01:48.800526 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:48.800616 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":308,"threads_started":1}
I20260925 15:01:49.242991 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:49.242127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:35799)
0925 15:01:49.242544 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:49.242819 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260925 15:01:49.345633 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:49.344944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:53551)
0925 15:01:49.345400 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:49.345483 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260925 15:01:49.802416 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:49.801731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:48211)
0925 15:01:49.802179 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:49.802263 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260925 15:01:50.244575 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:50.243893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:34991)
0925 15:01:50.244352 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:50.244429 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":259,"threads_started":1}
I20260925 15:01:50.347342 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:50.346535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:40963)
0925 15:01:50.347045 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:50.347153 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":296,"threads_started":1}
I20260925 15:01:50.804128 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:50.803364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:45807)
0925 15:01:50.803860 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:50.803950 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":254,"threads_started":1}
I20260925 15:01:51.246170 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:51.245448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:44557)
0925 15:01:51.245912 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:51.245996 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":216,"threads_started":1}
I20260925 15:01:51.348898 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:51.348241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:36253)
0925 15:01:51.348649 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:51.348725 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":202,"threads_started":1}
I20260925 15:01:51.805647 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:51.805025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:48561)
0925 15:01:51.805404 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:51.805479 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":192,"threads_started":1}
I20260925 15:01:52.247682 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:52.247042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:60497)
0925 15:01:52.247438 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:52.247514 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":206,"threads_started":1}
I20260925 15:01:52.350571 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:52.349936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.193:51333)
0925 15:01:52.350325 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:52.350413 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":197,"threads_started":1}
I20260925 15:01:52.807482 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:52.806762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.194:60135)
0925 15:01:52.807201 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:52.807307 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
I20260925 15:01:53.226663 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 20041
I20260925 15:01:53.245359 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:53.227707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.4.243.193:42905)
0925 15:01:53.228358 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:53.228377 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:53.228421 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:01:53.230997 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:53.231007 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:53.231438 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:53.231752 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:53.233316 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:53.234572 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:53.235780 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:53.236418 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:53.236548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:53.239121 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:53.239128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:53.239143 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:53.239181 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:53.242148 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:53.242910 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:53.242917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:53.242922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:53.243071 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:53.243455 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:53.243458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:53.243460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:53.243971 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:53.244191 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:53.244566 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 15:01:53.245020 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":326,"threads_started":1}
I20260925 15:01:53.246752 20199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:53.252060 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:53.248957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36395 (local address 127.4.243.195:58497)
0925 15:01:53.251797 (+ 2840us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:53.251888 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:36395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2705,"thread_start_us":2590,"threads_started":1}
I20260925 15:01:53.331192 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 20041
I20260925 15:01:53.348623 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 20207
I20260925 15:01:53.366218 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:53.349531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.4.243.194:32843)
0925 15:01:53.350051 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:53.350066 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:53.350101 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:01:53.351688 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:53.351698 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:53.352115 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:53.352465 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:53.353963 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:53.355274 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:53.356506 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:53.357161 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:53.357362 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:53.360167 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:53.360174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:53.360188 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:53.360226 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:53.363014 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:53.363684 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:53.363693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:53.363707 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:53.363887 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:53.364330 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:53.364337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:53.364344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:53.364807 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:53.365095 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:53.365547 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 15:01:53.365950 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
I20260925 15:01:53.367578 20330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:53.451267 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 20207
I20260925 15:01:53.470263 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 20338
I20260925 15:01:53.487666 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:53.471263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.4.243.195:34517)
0925 15:01:53.471869 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:53.471883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:53.471925 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:01:53.473528 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:53.473537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:53.473958 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:53.474245 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:53.475726 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:53.476888 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:53.478160 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:53.478898 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:53.479035 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:53.481675 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:53.481683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:53.481697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:53.481729 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:53.484426 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:53.485129 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:53.485135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:53.485140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:53.485312 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:53.485651 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:53.485654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:53.485656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:53.486150 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:53.486391 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:53.487078 (+ 687us) server_negotiation.cc:299] Negotiation successful
0925 15:01:53.487389 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":301,"threads_started":1}
I20260925 15:01:53.489073 20461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:53.570654 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 20338
2026-09-25T15:01:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24021 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:01:53 dist-test-slave-qxh0 krb5kdc[20523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:01:53 dist-test-slave-qxh0 krb5kdc[20523](info): set up 2 sockets
Sep 25 15:01:53 dist-test-slave-qxh0 krb5kdc[20523](info): commencing operation
krb5kdc: starting...
W20260925 15:01:54.655200 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.015s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:01:54 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:01:54.887799 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33587
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:55.210574 20536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:55.210913 20536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:55.211071 20536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:55.218353 20536 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:01:55.218451 20536 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:01:55.218518 20536 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:01:55.218608 20536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:55.218727 20536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:55.218839 20536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:55.218909 20536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:55.218986 20536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:55.228896 20536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:33587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:55.230176 20536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:55.231527 20536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:55.243794 20541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.243764 20544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.243815 20542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:55.244378 20536 server_base.cc:1034] running on GCE node
I20260925 15:01:55.245030 20536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:55.246320 20536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:55.247558 20536 hybrid_clock.cc:648] HybridClock initialized: now 1790348515247504 us; error 33 us; skew 500 ppm
I20260925 15:01:55.247844 20536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:55 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:55.257478 20536 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:01:55.259207 20536 webserver.cc:460] Webserver started at http://127.4.243.254:37439/ using document root <none> and password file <none>
I20260925 15:01:55.259965 20536 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:55.260057 20536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:55.260403 20536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:55.263800 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "1b479d2b57a04275afa44e011012af33"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "65db8d5027988d3a0c7f847719aef87e"
server_key_iv: "8af2dd513c34d2150c0b11cfdd3c6809"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.264586 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "1b479d2b57a04275afa44e011012af33"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "65db8d5027988d3a0c7f847719aef87e"
server_key_iv: "8af2dd513c34d2150c0b11cfdd3c6809"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.269616 20536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:01:55.273043 20551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.274284 20536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:01:55.274456 20536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "1b479d2b57a04275afa44e011012af33"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "65db8d5027988d3a0c7f847719aef87e"
server_key_iv: "8af2dd513c34d2150c0b11cfdd3c6809"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.274698 20536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:55.291024 20536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:55.294837 20536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:55.295095 20536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:55.313141 20536 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:33587
I20260925 15:01:55.313148 20602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:33587 every 8 connection(s)
I20260925 15:01:55.314419 20536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:55.317893 20603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:55.319689 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 20536
I20260925 15:01:55.319904 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:01:55.320520 5071 external_mini_cluster.cc:1428] Setting key 4ff1a77a0db2a7102655ae5d3384d254
I20260925 15:01:55.330475 20603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Bootstrap starting.
I20260925 15:01:55.333983 20603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:55.335062 20603 log.cc:824] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:01:55 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:55.338243 20603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: No bootstrap required, opened a new log
I20260925 15:01:55.344532 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.322424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.4.243.254:33587)
0925 15:01:55.323310 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.323324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.323372 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:01:55.325695 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.325731 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.326220 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.326657 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:55.328011 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.329881 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.331293 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.332337 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.332525 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.337162 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:55.337170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:55.337186 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:55.337236 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:55.340453 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:55.341272 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:55.341278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:55.341285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:55.341469 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:55.341866 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:55.341869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:55.341871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:55.342434 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:55.342747 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.343141 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.343514 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":397,"threads_started":1}
I20260925 15:01:55.347559 20603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:01:55.347941 20603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:55.348039 20603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b479d2b57a04275afa44e011012af33, State: Initialized, Role: FOLLOWER
I20260925 15:01:55.348598 20603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:01:55.348839 20603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:55.348969 20603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:55.349156 20603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:55.351455 20603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:01:55.351923 20603 leader_election.cc:302] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b479d2b57a04275afa44e011012af33; no voters:
I20260925 15:01:55.352484 20603 leader_election.cc:288] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:55.352612 20608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:55.353307 20608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 LEADER]: Becoming Leader. State: Replica: 1b479d2b57a04275afa44e011012af33, State: Running, Role: LEADER
I20260925 15:01:55.353753 20608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:01:55.354689 20603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:55.356781 20610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b479d2b57a04275afa44e011012af33. Latest consensus state: current_term: 1 leader_uuid: "1b479d2b57a04275afa44e011012af33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } } }
I20260925 15:01:55.357188 20610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:55.358363 20609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b479d2b57a04275afa44e011012af33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } } }
I20260925 15:01:55.358750 20609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:55.359157 20617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:55.363222 20617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:55.370924 20617 catalog_manager.cc:1347] Generated new cluster ID: ae6f3436a31844fd9273e3fba9a4dc5d
I20260925 15:01:55.371031 20617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:55.388451 20617 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:55.389669 20617 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:55.401276 20617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Generated new TSK 0
I20260925 15:01:55.402056 20617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 15:01:55.458977 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33587
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:55.778565 20629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:55.779055 20629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:55.779224 20629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:55.786376 20629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:55.786525 20629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:55.786726 20629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:01:55.796123 20629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:55.797401 20629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:55.798712 20629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:55.811247 20637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.811287 20634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.811324 20635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:55.812268 20629 server_base.cc:1034] running on GCE node
I20260925 15:01:55.812796 20629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:55.813653 20629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:55.814860 20629 hybrid_clock.cc:648] HybridClock initialized: now 1790348515814823 us; error 27 us; skew 500 ppm
I20260925 15:01:55.815145 20629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:55 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:55.818178 20629 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:01:55.819578 20629 webserver.cc:460] Webserver started at http://127.4.243.193:37651/ using document root <none> and password file <none>
I20260925 15:01:55.820242 20629 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:55.820343 20629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:55.820597 20629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:55.823465 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "1bfb15597ec14588817ce424a9412ace"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "9c3354967ce2a3a0bf2004e3c7a8132e"
server_key_iv: "e764ac81ef7adee7c98d7c7655ede8ef"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.824154 20629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "1bfb15597ec14588817ce424a9412ace"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "9c3354967ce2a3a0bf2004e3c7a8132e"
server_key_iv: "e764ac81ef7adee7c98d7c7655ede8ef"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.828994 20629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 15:01:55.832242 20644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.833487 20629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:01:55.833606 20629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "1bfb15597ec14588817ce424a9412ace"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "9c3354967ce2a3a0bf2004e3c7a8132e"
server_key_iv: "e764ac81ef7adee7c98d7c7655ede8ef"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.833779 20629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:55.849028 20629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:55.852789 20629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:55.853009 20629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:55.853894 20629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:55.855396 20629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:55.855471 20629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.855661 20629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:55.855741 20629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.889174 20629 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:37003
I20260925 15:01:55.889212 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:37003 every 8 connection(s)
I20260925 15:01:55.890403 20629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:55.892584 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 20629
I20260925 15:01:55.892791 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:55.893602 5071 external_mini_cluster.cc:1428] Setting key b6197ebc56c8898a950a2ec9ed823904
Sep 25 15:01:55 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348515, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:55.911118 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.893207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:48439 (local address 127.4.243.254:33587)
0925 15:01:55.893697 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.893702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.894815 (+ 1113us) server_negotiation.cc:407] Connection header received
0925 15:01:55.896746 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.896753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.896846 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.896966 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:55.898468 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.899675 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.901270 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.902002 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.902131 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.905650 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:55.905656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:55.905667 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:55.905704 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:55.908223 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:55.908927 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:55.908933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:55.908943 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:55.909050 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:55.909454 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:55.909460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:55.909463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:55.909737 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:55.909887 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.910523 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.910763 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260925 15:01:55.912082 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.893798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:48439)
0925 15:01:55.894653 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.894703 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.896472 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.897170 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.897179 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.897704 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:55.898298 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.898316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.899830 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.899834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.901127 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.901138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.902214 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.902227 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.902420 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.903167 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:55.903232 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:55.905433 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:55.908366 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:55.908374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:55.908390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:55.908778 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:55.909154 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:55.909159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:55.909163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:55.909347 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:55.909872 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:55.909879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:55.910188 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.910455 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.910959 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":372,"threads_started":1}
I20260925 15:01:55.913792 20757 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33587
I20260925 15:01:55.914176 20757 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:55.914980 20757 heartbeater.cc:511] Master 127.4.243.254:33587 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 15:01:55.917177 20568 ts_manager.cc:194] Registered new tserver with Master: 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
I20260925 15:01:55.918666 20568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:48439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 15:01:55.946007 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33587
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:56.260305 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:56.260667 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:56.260848 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:56.267836 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:56.267997 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:56.268184 20763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:01:56.278002 20763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:56.279273 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:56.280519 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:56.292775 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.292745 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.293228 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:56.293533 20763 server_base.cc:1034] running on GCE node
I20260925 15:01:56.294107 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:56.295140 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:56.296388 20763 hybrid_clock.cc:648] HybridClock initialized: now 1790348516296340 us; error 47 us; skew 500 ppm
I20260925 15:01:56.296707 20763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:56 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:56.299870 20763 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:01:56.301234 20763 webserver.cc:460] Webserver started at http://127.4.243.194:35943/ using document root <none> and password file <none>
I20260925 15:01:56.301880 20763 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:56.302021 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:56.302320 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:56.305176 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "6274738a93c54ace9f752d91c7120d74"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "b536761c98d6969bf30b94ff7b4c6f14"
server_key_iv: "8cb339c02964c2005111f36e6abe37f6"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.305920 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "6274738a93c54ace9f752d91c7120d74"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "b536761c98d6969bf30b94ff7b4c6f14"
server_key_iv: "8cb339c02964c2005111f36e6abe37f6"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.310808 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:01:56.314088 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.315377 20763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:56.315539 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "6274738a93c54ace9f752d91c7120d74"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "b536761c98d6969bf30b94ff7b4c6f14"
server_key_iv: "8cb339c02964c2005111f36e6abe37f6"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.315726 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:56.329121 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:56.332950 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:56.333217 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:56.334147 20763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:56.335568 20763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:56.335695 20763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.335846 20763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:56.335927 20763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.367887 20763 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:42015
I20260925 15:01:56.367928 20890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:42015 every 8 connection(s)
I20260925 15:01:56.369072 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:56.377823 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 20763
I20260925 15:01:56.378017 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:56.378453 5071 external_mini_cluster.cc:1428] Setting key 9f1c5c36b2fcbcb1d921bed55166453e
Sep 25 15:01:56 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:56.390578 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.371645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:42515 (local address 127.4.243.254:33587)
0925 15:01:56.371858 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.371863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.372893 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 15:01:56.374705 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.374711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.374808 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.374927 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:56.376440 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.377179 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.379182 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.379800 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.379904 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.383790 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:56.383798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:56.383806 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:56.383850 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:56.387565 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.388245 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.388252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.388258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.388410 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.388799 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.388806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.388812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.389174 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:56.389292 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.390186 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.390365 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 15:01:56.392032 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.372046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:42515)
0925 15:01:56.372738 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:56.372773 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:56.374394 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:56.375199 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:56.375208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:56.375697 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:56.376264 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.376281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.377375 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.377380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.379001 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.379014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.380031 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.380038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.380343 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.381140 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:56.381161 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:56.383566 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:56.387694 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.387701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.387712 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.388120 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.388505 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.388507 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.388509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.388672 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.389299 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:56.389309 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:56.389752 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 15:01:56.390068 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 15:01:56.390643 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":277,"threads_started":1}
I20260925 15:01:56.393425 20891 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33587
I20260925 15:01:56.393723 20891 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:56.394428 20891 heartbeater.cc:511] Master 127.4.243.254:33587 requested a full tablet report, sending...
I20260925 15:01:56.395879 20568 ts_manager.cc:194] Registered new tserver with Master: 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015)
I20260925 15:01:56.396641 20568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:42515
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 15:01:56.431545 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33587
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:56.747375 20896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:56.747681 20896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:56.747861 20896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:56.755009 20896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:56.755136 20896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:56.755306 20896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:01:56.765411 20896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:01:56.766618 20896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:56.767880 20896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:56.780711 20904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.780676 20902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.780766 20901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:56.781517 20896 server_base.cc:1034] running on GCE node
I20260925 15:01:56.782102 20896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:56.783082 20896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:56.784305 20896 hybrid_clock.cc:648] HybridClock initialized: now 1790348516784269 us; error 49 us; skew 500 ppm
I20260925 15:01:56.784647 20896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:01:56 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:56.787649 20896 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:01:56.788973 20896 webserver.cc:460] Webserver started at http://127.4.243.195:35467/ using document root <none> and password file <none>
I20260925 15:01:56.789616 20896 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:56.789760 20896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:56.790055 20896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:56.792796 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "2b10ea8147bf48ffbea9d544a945c8e9"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "853fe3e46dafd1f941a722e53da1b23c"
server_key_iv: "da9a5adecab05a31474747d14eea6815"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.793503 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "2b10ea8147bf48ffbea9d544a945c8e9"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "853fe3e46dafd1f941a722e53da1b23c"
server_key_iv: "da9a5adecab05a31474747d14eea6815"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.798179 20896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:01:56.801270 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.802469 20896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:56.802683 20896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "2b10ea8147bf48ffbea9d544a945c8e9"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "853fe3e46dafd1f941a722e53da1b23c"
server_key_iv: "da9a5adecab05a31474747d14eea6815"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.802865 20896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:56.833711 20896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:56.838172 20896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:56.838471 20896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:56.839478 20896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:56.840801 20896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:56.840898 20896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.841034 20896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:56.841111 20896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.872973 20896 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42363
I20260925 15:01:56.873018 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42363 every 8 connection(s)
I20260925 15:01:56.874264 20896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:56.876060 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 20896
I20260925 15:01:56.876250 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:56.876775 5071 external_mini_cluster.cc:1428] Setting key af15c9ce4785fbd36b8d08cf178b9816
Sep 25 15:01:56 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:01:56.893517 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.876771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:53553 (local address 127.4.243.254:33587)
0925 15:01:56.876987 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.876993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.878325 (+ 1332us) server_negotiation.cc:407] Connection header received
0925 15:01:56.880123 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.880127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.880186 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.880289 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:56.881605 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.882282 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.883696 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.884098 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.884189 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.887387 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:56.887391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:56.887394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:56.887421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:56.890241 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.891208 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.891212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.891214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.891329 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.891781 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.891785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.891789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.892125 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:56.892224 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.892967 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.893185 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:01:56.894726 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.877299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:53553)
0925 15:01:56.878165 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:56.878208 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:56.879907 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:56.880478 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:56.880485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:56.880962 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:56.881476 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.881492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.882410 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.882413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.883585 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.883595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.884267 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.884272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.884457 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.885117 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:56.885138 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:56.887219 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:56.890374 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.890382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.890395 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.890961 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.891443 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.891448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.891452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.891658 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.892327 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:56.892334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:56.892640 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 15:01:56.892875 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 15:01:56.893354 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":375,"threads_started":1}
I20260925 15:01:56.896068 21024 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33587
I20260925 15:01:56.896324 21024 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:56.896963 21024 heartbeater.cc:511] Master 127.4.243.254:33587 requested a full tablet report, sending...
I20260925 15:01:56.898265 20568 ts_manager.cc:194] Registered new tserver with Master: 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363)
I20260925 15:01:56.899101 20568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:53553
I20260925 15:01:56.904086 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:56.920646 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.908300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.4.243.254:33587)
0925 15:01:56.908447 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.908452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.908910 (+ 458us) server_negotiation.cc:407] Connection header received
0925 15:01:56.909037 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.909040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.909093 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.909183 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:56.910503 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.911300 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.912647 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.913046 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.913135 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.915541 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:56.915546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:56.915549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:56.915576 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:56.918034 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.918967 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.918971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.918974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.919114 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.919449 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.919452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.919454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.919776 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:56.919953 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.920290 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.920441 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:01:56.922555 20757 heartbeater.cc:503] Master 127.4.243.254:33587 was elected leader, sending a full tablet report...
I20260925 15:01:56.924388 20567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:56.931102 20567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:56.956646 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.943091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58688)
0925 15:01:56.944135 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:56.944189 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:56.944372 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:56.946146 (+ 1774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:56.946154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:56.946191 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:56.946771 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.946790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.949474 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.949483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.952739 (+ 3256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.952764 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.955313 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.955323 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.955631 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.955736 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 15:01:56.955896 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 15:01:56.956138 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":56,"thread_start_us":919,"threads_started":1}
I20260925 15:01:56.958076 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.943399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.4.243.193:37003)
0925 15:01:56.945473 (+ 2074us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.945483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.945510 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:01:56.945600 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.945609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.945897 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.947029 (+ 1132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:56.947322 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.949293 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.953023 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.955014 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.957208 (+ 2194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.957454 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.957528 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.957721 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1929,"thread_start_us":1802,"threads_started":1}
I20260925 15:01:56.958873 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.943456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42015 (local address 127.0.0.1:43112)
0925 15:01:56.945029 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:56.945047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:56.945169 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:56.947569 (+ 2400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:56.947576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:56.947608 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:56.948120 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.948134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.951703 (+ 3569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.951711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.955865 (+ 4154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.955884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.957970 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.957976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.958166 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.958211 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:01:56.958309 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:01:56.958443 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"mutex_wait_us":4,"thread_start_us":2645,"threads_started":1}
I20260925 15:01:56.958920 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.943787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.4.243.194:42015)
0925 15:01:56.944989 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.945000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.945314 (+ 314us) server_negotiation.cc:407] Connection header received
0925 15:01:56.945436 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.945447 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.945782 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.946001 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:56.948362 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.951512 (+ 3150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.956090 (+ 4578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.957772 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.957912 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.958144 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.958377 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.958586 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":898,"threads_started":1}
I20260925 15:01:56.961715 20692 tablet_service.cc:1467] Processing CreateTablet for tablet 0faff72aef58487ea42c3692ecd2091a (DEFAULT_TABLE table=test-table [id=add74cf5717d4ce89cec2dd5e8d4d964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:56.963135 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faff72aef58487ea42c3692ecd2091a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:56.964083 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.945345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42363 (local address 127.0.0.1:40004)
0925 15:01:56.957442 (+ 12097us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:56.957460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:56.957583 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:56.958452 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:56.958485 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:56.958509 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:56.958872 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.958883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.960591 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.960599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.961837 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.961848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.963293 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.963299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.963480 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.963515 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:01:56.963627 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:01:56.963747 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11950,"thread_start_us":11667,"threads_started":1}
I20260925 15:01:56.964149 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.945839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.4.243.195:42363)
0925 15:01:56.956699 (+ 10860us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.956708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.957569 (+ 861us) server_negotiation.cc:407] Connection header received
0925 15:01:56.957725 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.957734 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.957994 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.958183 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:56.959082 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.960459 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.961991 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.963110 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.963249 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.963458 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.963673 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.963811 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10678,"thread_start_us":10860,"threads_started":1}
I20260925 15:01:56.965879 20826 tablet_service.cc:1467] Processing CreateTablet for tablet 0faff72aef58487ea42c3692ecd2091a (DEFAULT_TABLE table=test-table [id=add74cf5717d4ce89cec2dd5e8d4d964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:56.967254 20826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faff72aef58487ea42c3692ecd2091a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:56.967123 20959 tablet_service.cc:1467] Processing CreateTablet for tablet 0faff72aef58487ea42c3692ecd2091a (DEFAULT_TABLE table=test-table [id=add74cf5717d4ce89cec2dd5e8d4d964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:56.968212 20959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faff72aef58487ea42c3692ecd2091a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:56.979631 21042 tablet_bootstrap.cc:492] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Bootstrap starting.
I20260925 15:01:56.979946 21043 tablet_bootstrap.cc:492] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Bootstrap starting.
I20260925 15:01:56.981310 21044 tablet_bootstrap.cc:492] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Bootstrap starting.
I20260925 15:01:56.983317 21043 tablet_bootstrap.cc:654] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:56.983549 21042 tablet_bootstrap.cc:654] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:56.984409 21043 log.cc:824] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:56.984550 21044 tablet_bootstrap.cc:654] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:56.984787 21042 log.cc:824] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:56.985773 21044 log.cc:824] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:56.987608 21043 tablet_bootstrap.cc:492] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: No bootstrap required, opened a new log
I20260925 15:01:56.987864 21043 ts_tablet_manager.cc:1397] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260925 15:01:56.987900 21042 tablet_bootstrap.cc:492] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: No bootstrap required, opened a new log
I20260925 15:01:56.988137 21042 ts_tablet_manager.cc:1397] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 15:01:56.988562 21044 tablet_bootstrap.cc:492] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: No bootstrap required, opened a new log
I20260925 15:01:56.988792 21044 ts_tablet_manager.cc:1397] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260925 15:01:56.994699 21043 raft_consensus.cc:348] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:56.995036 21043 raft_consensus.cc:374] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:56.995177 21043 raft_consensus.cc:729] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b10ea8147bf48ffbea9d544a945c8e9, State: Initialized, Role: FOLLOWER
I20260925 15:01:56.995728 21043 consensus_queue.cc:260] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:56.996734 21024 heartbeater.cc:503] Master 127.4.243.254:33587 was elected leader, sending a full tablet report...
I20260925 15:01:56.997359 21043 ts_tablet_manager.cc:1428] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20260925 15:01:56.998636 21044 raft_consensus.cc:348] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:56.999034 21044 raft_consensus.cc:374] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:56.999130 21044 raft_consensus.cc:729] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6274738a93c54ace9f752d91c7120d74, State: Initialized, Role: FOLLOWER
I20260925 15:01:56.999084 21042 raft_consensus.cc:348] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:56.999521 21042 raft_consensus.cc:374] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:56.999630 21042 raft_consensus.cc:729] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfb15597ec14588817ce424a9412ace, State: Initialized, Role: FOLLOWER
I20260925 15:01:56.999691 21044 consensus_queue.cc:260] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:57.000134 21042 consensus_queue.cc:260] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:57.000803 20891 heartbeater.cc:503] Master 127.4.243.254:33587 was elected leader, sending a full tablet report...
I20260925 15:01:57.001307 21044 ts_tablet_manager.cc:1428] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 15:01:57.002444 21042 ts_tablet_manager.cc:1428] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 15:01:57.041435 21048 raft_consensus.cc:482] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:57.041730 21048 raft_consensus.cc:504] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:57.043371 21048 leader_election.cc:288] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003), 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015)
I20260925 15:01:57.049681 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.043799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:36703 (local address 127.4.243.194:42015)
0925 15:01:57.044011 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.044017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.044332 (+ 315us) server_negotiation.cc:407] Connection header received
0925 15:01:57.044539 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.044545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.044624 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.044739 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.045180 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.046219 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.048094 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.048840 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.049012 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.049077 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.049319 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.049432 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:01:57.049921 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.043857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:39385 (local address 127.4.243.193:37003)
0925 15:01:57.044075 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.044087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.044758 (+ 671us) server_negotiation.cc:407] Connection header received
0925 15:01:57.044903 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.044908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.044982 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.045076 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.045738 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.046659 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.048354 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.049083 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.049253 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.049319 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.049567 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.049702 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:01:57.049736 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.043805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42015 (local address 127.4.243.195:36703)
0925 15:01:57.044199 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.044226 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.044394 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.044807 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.044812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.044843 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.045036 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.045052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.046342 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.046358 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.047947 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.047958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.048995 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.049011 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.049172 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.049193 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.049297 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.049427 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":33}
I20260925 15:01:57.050311 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.043885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.4.243.195:39385)
0925 15:01:57.044644 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.044662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.044775 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.045093 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.045096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.045135 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.045525 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.045539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.046810 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.046815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.048198 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.048209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.049277 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.049283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.049420 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.049441 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.049534 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.049666 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":413,"threads_started":1}
I20260925 15:01:57.051071 20846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faff72aef58487ea42c3692ecd2091a" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6274738a93c54ace9f752d91c7120d74" is_pre_election: true
I20260925 15:01:57.051401 20712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faff72aef58487ea42c3692ecd2091a" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfb15597ec14588817ce424a9412ace" is_pre_election: true
I20260925 15:01:57.051635 20846 raft_consensus.cc:2454] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 0.
I20260925 15:01:57.051913 20712 raft_consensus.cc:2454] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 0.
I20260925 15:01:57.052341 20914 leader_election.cc:302] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74; no voters:
I20260925 15:01:57.052708 21048 raft_consensus.cc:2781] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:57.052846 21048 raft_consensus.cc:482] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:57.052942 21048 raft_consensus.cc:3037] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.055295 21048 raft_consensus.cc:504] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:57.056012 21048 leader_election.cc:288] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 election: Requested vote from peers 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003), 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015)
I20260925 15:01:57.056448 20712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faff72aef58487ea42c3692ecd2091a" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfb15597ec14588817ce424a9412ace"
I20260925 15:01:57.056548 20846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faff72aef58487ea42c3692ecd2091a" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6274738a93c54ace9f752d91c7120d74"
I20260925 15:01:57.056676 20712 raft_consensus.cc:3037] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.056759 20846 raft_consensus.cc:3037] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.059957 20846 raft_consensus.cc:2454] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 1.
I20260925 15:01:57.059957 20712 raft_consensus.cc:2454] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 1.
I20260925 15:01:57.060367 20914 leader_election.cc:302] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74; no voters:
I20260925 15:01:57.060621 21048 raft_consensus.cc:2781] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:57.061094 21048 raft_consensus.cc:686] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 2b10ea8147bf48ffbea9d544a945c8e9, State: Running, Role: LEADER
I20260925 15:01:57.061590 21048 consensus_queue.cc:237] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:01:57.065893 20568 catalog_manager.cc:5495] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } health_report { overall_health: HEALTHY } } }
I20260925 15:01:57.081594 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.076443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.4.243.195:42363)
0925 15:01:57.076624 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.076632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.076668 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:01:57.076922 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.076926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.076994 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.077136 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:57.077482 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.078204 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.079644 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.080028 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.080150 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.080608 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:57.081047 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:57.081193 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.081284 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.081365 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:01:57.088804 20712 raft_consensus.cc:1261] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Refusing update from remote peer 2b10ea8147bf48ffbea9d544a945c8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:57.088831 20846 raft_consensus.cc:1261] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Refusing update from remote peer 2b10ea8147bf48ffbea9d544a945c8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:57.089661 21048 consensus_queue.cc:1035] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:57.089958 21052 consensus_queue.cc:1035] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:57.101279 21055 mvcc.cc:205] Tried to move back new op lower bound from 7333267525987172352 to 7333267525887881216. Current Snapshot: MvccSnapshot[applied={T|T < 7333267525987172352}]
I20260925 15:01:57.113938 20567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:57.114228 20567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:57.117635 20567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Sending DeleteTablet for 3 replicas of tablet 0faff72aef58487ea42c3692ecd2091a
I20260925 15:01:57.118747 20826 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faff72aef58487ea42c3692ecd2091a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:43112
I20260925 15:01:57.118744 20692 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faff72aef58487ea42c3692ecd2091a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58688
I20260925 15:01:57.118819 20959 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faff72aef58487ea42c3692ecd2091a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:40004
I20260925 15:01:57.119957 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 20629
I20260925 15:01:57.132530 21066 tablet_replica.cc:330] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: stopping tablet replica
I20260925 15:01:57.133275 21067 tablet_replica.cc:330] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: stopping tablet replica
I20260925 15:01:57.134004 21067 raft_consensus.cc:2229] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:57.134660 21066 raft_consensus.cc:2229] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:57.135385 21066 raft_consensus.cc:2258] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:57.136200 21067 raft_consensus.cc:2258] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:57.140228 21067 ts_tablet_manager.cc:1905] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:57.140905 21065 tablet_replica.cc:330] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: stopping tablet replica
I20260925 15:01:57.141201 21066 ts_tablet_manager.cc:1905] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:57.141393 21065 raft_consensus.cc:2229] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:57.142906 21065 raft_consensus.cc:2258] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 15:01:57 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 15:01:57.148751 21066 ts_tablet_manager.cc:1918] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:57.148988 21066 log.cc:1196] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/0faff72aef58487ea42c3692ecd2091a
I20260925 15:01:57.149446 21067 ts_tablet_manager.cc:1918] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:57.149526 21066 ts_tablet_manager.cc:1939] T 0faff72aef58487ea42c3692ecd2091a P 6274738a93c54ace9f752d91c7120d74: Deleting consensus metadata
I20260925 15:01:57.149773 21067 log.cc:1196] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/0faff72aef58487ea42c3692ecd2091a
I20260925 15:01:57.151180 21065 ts_tablet_manager.cc:1905] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:57.151853 20554 catalog_manager.cc:4841] TS 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015): tablet 0faff72aef58487ea42c3692ecd2091a (table test-table [id=add74cf5717d4ce89cec2dd5e8d4d964]) successfully deleted
I20260925 15:01:57.152616 21067 ts_tablet_manager.cc:1939] T 0faff72aef58487ea42c3692ecd2091a P 2b10ea8147bf48ffbea9d544a945c8e9: Deleting consensus metadata
I20260925 15:01:57.154209 20555 catalog_manager.cc:4841] TS 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363): tablet 0faff72aef58487ea42c3692ecd2091a (table test-table [id=add74cf5717d4ce89cec2dd5e8d4d964]) successfully deleted
I20260925 15:01:57.157042 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.121051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.4.243.193:37003)
0925 15:01:57.121220 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.121225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.137346 (+ 16121us) server_negotiation.cc:407] Connection header received
0925 15:01:57.137498 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.137503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.137589 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.137684 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.139734 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.140980 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.143098 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.143732 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.143827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.149495 (+ 5668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.149509 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.149526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:57.149566 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.153464 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.154581 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.154619 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.154623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.154772 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.155257 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.155262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.155266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.155733 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.156058 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.156380 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.156632 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:01:57.157887 20752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:57.243446 21065 ts_tablet_manager.cc:1918] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:57.243809 21065 log.cc:1196] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/0faff72aef58487ea42c3692ecd2091a
I20260925 15:01:57.244091 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 20629
W20260925 15:01:57.267474 20553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:37003 (error 108)
I20260925 15:01:57.267778 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 20763
I20260925 15:01:57.269819 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.269325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58702)
0925 15:01:57.269504 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.269619 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 15:01:57.271183 20553 catalog_manager.cc:4584] TS 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0faff72aef58487ea42c3692ecd2091a: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Sep 25 15:01:57 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:01:57.285251 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.268974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.4.243.194:42015)
0925 15:01:57.269149 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.269156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.269174 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:01:57.269254 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.269260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.269339 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.269452 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.271134 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.272277 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.273679 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.274317 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.274456 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.278841 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.278854 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.278872 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:57.278917 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.282407 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.283167 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.283177 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.283180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.283329 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.283702 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.283707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.283711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.284152 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.284460 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.284676 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.284885 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:01:57.286023 20886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:57.324986 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.324442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58706)
0925 15:01:57.324704 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.324807 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 15:01:57.326205 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.325816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58718)
0925 15:01:57.325991 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.326063 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 15:01:57.375857 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 20763
I20260925 15:01:57.386790 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.386274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58728)
0925 15:01:57.386484 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.386564 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:01:57.387799 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.387462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58740)
0925 15:01:57.387612 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.387680 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:01:57.397924 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 20896
Sep 25 15:01:57 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 15:01:57.416006 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.398780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.4.243.195:42363)
0925 15:01:57.398937 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.398941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.398958 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:01:57.399102 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.399105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.399173 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.399265 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.400918 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.401802 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.403647 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.404077 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.404190 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.409877 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.409888 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.409902 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:57.409935 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.412898 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.413628 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.413634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.413637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.413815 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.414262 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.414266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.414268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.414682 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.415011 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.415350 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.415597 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:01:57.416731 21019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:57.468357 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.467863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58754)
0925 15:01:57.468093 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.468192 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 15:01:57.469480 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.469071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58766)
0925 15:01:57.469286 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.469369 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260925 15:01:57.541653 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 20896
I20260925 15:01:57.562273 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 20536
I20260925 15:01:57.563478 20598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:01:57.672425 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.671521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:58770)
0925 15:01:57.672115 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.672203 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
I20260925 15:01:57.672509 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 20536
I20260925 15:02:17.689913 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33587
--webserver_interface=127.4.243.254
--webserver_port=37439
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:18.012562 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:18.012821 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:18.012977 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:18.019976 21074 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:02:18.020071 21074 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:02:18.020141 21074 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:02:18.020237 21074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:18.020329 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:18.020462 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:18.020534 21074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:18.020612 21074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:18.030301 21074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:33587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=37439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:18.031477 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:18.032725 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:18.054394 21081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:18.054355 21083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:18.054435 21080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:18.054639 21074 server_base.cc:1034] running on GCE node
I20260925 15:02:18.055750 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:18.057246 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:18.058504 21074 hybrid_clock.cc:648] HybridClock initialized: now 1790348538058445 us; error 51 us; skew 500 ppm
I20260925 15:02:18.058938 21074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:18 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:18.066063 21074 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:02:18.067770 21074 webserver.cc:460] Webserver started at http://127.4.243.254:37439/ using document root <none> and password file <none>
I20260925 15:02:18.068488 21074 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:18.068585 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:18.074470 21074 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 15:02:18.077389 21090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:18.078946 21074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:02:18.079133 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "1b479d2b57a04275afa44e011012af33"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "65db8d5027988d3a0c7f847719aef87e"
server_key_iv: "8af2dd513c34d2150c0b11cfdd3c6809"
server_key_version: "encryptionkey@0"
I20260925 15:02:18.080166 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:18.103363 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:18.107712 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:18.108042 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:18.127260 21074 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:33587
I20260925 15:02:18.127298 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:33587 every 8 connection(s)
I20260925 15:02:18.128767 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:18.132790 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21074
I20260925 15:02:18.133816 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:37003
--local_ip_for_outbound_sockets=127.4.243.193
--tserver_master_addrs=127.4.243.254:33587
--webserver_port=37651
--webserver_interface=127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:02:18.137111 21142 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:02:18.142515 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Bootstrap starting.
I20260925 15:02:18.162276 21142 log.cc:824] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:18.173939 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:02:18.174557 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Bootstrap complete.
I20260925 15:02:18.183880 21142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:18.184875 21142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b479d2b57a04275afa44e011012af33, State: Initialized, Role: FOLLOWER
I20260925 15:02:18.185549 21142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:18.185848 21142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:18.185967 21142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:18.186131 21142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:02:18.190119 21142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:18.190725 21142 leader_election.cc:302] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b479d2b57a04275afa44e011012af33; no voters:
I20260925 15:02:18.191377 21142 leader_election.cc:288] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:02:18.191509 21146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:02:18.192718 21146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 LEADER]: Becoming Leader. State: Replica: 1b479d2b57a04275afa44e011012af33, State: Running, Role: LEADER
I20260925 15:02:18.193389 21146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:18.193794 21142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:18.196162 21147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b479d2b57a04275afa44e011012af33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } } }
I20260925 15:02:18.196564 21147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:18.196157 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b479d2b57a04275afa44e011012af33. Latest consensus state: current_term: 2 leader_uuid: "1b479d2b57a04275afa44e011012af33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } } }
I20260925 15:02:18.196926 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:18.198084 21153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:18.217860 21153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:18.220158 21153 catalog_manager.cc:1259] Loaded cluster ID: ae6f3436a31844fd9273e3fba9a4dc5d
I20260925 15:02:18.220439 21153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:18.226251 21153 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:18.228530 21153 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Loaded TSK: 0
I20260925 15:02:18.242987 21163 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:02:18.260399 21153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Generated new TSK 1
I20260925 15:02:18.260807 21153 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:02:18.498065 21144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:18.498327 21144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:18.498473 21144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:18.506011 21144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:18.506124 21144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:18.506319 21144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:02:18.516681 21144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:37003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=37651
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:18.517959 21144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:18.519300 21144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:18.533541 21169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:18.533572 21170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:18.534701 21172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:18.535411 21144 server_base.cc:1034] running on GCE node
I20260925 15:02:18.536099 21144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:18.537047 21144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:18.538298 21144 hybrid_clock.cc:648] HybridClock initialized: now 1790348538538232 us; error 56 us; skew 500 ppm
I20260925 15:02:18.538684 21144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:18 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:18.542166 21144 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:02:18.543993 21144 webserver.cc:460] Webserver started at http://127.4.243.193:37651/ using document root <none> and password file <none>
I20260925 15:02:18.544966 21144 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:18.545121 21144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:18.552557 21144 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:02:18.555855 21179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:18.557471 21144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:02:18.557634 21144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "1bfb15597ec14588817ce424a9412ace"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "9c3354967ce2a3a0bf2004e3c7a8132e"
server_key_iv: "e764ac81ef7adee7c98d7c7655ede8ef"
server_key_version: "encryptionkey@0"
I20260925 15:02:18.558952 21144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:18.584335 21144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:18.589601 21144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:18.589860 21144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:18.590981 21144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:18.593042 21186 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 15:02:18.598162 21186 ts_tablet_manager.cc:1852] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 15:02:18.599165 21186 ts_tablet_manager.cc:1905] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:02:18.605996 21186 ts_tablet_manager.cc:1918] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:02:18.606197 21186 log.cc:1196] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/0faff72aef58487ea42c3692ecd2091a
I20260925 15:02:18.606637 21186 ts_tablet_manager.cc:1939] T 0faff72aef58487ea42c3692ecd2091a P 1bfb15597ec14588817ce424a9412ace: Deleting consensus metadata
I20260925 15:02:18.607064 21144 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 15:02:18.607172 21144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260925 15:02:18.607280 21144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:18.607353 21144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:18.637832 21144 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:37003
I20260925 15:02:18.637869 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:37003 every 8 connection(s)
I20260925 15:02:18.639056 21144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:02:18.641211 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21144
I20260925 15:02:18.642005 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:42015
--local_ip_for_outbound_sockets=127.4.243.194
--tserver_master_addrs=127.4.243.254:33587
--webserver_port=35943
--webserver_interface=127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:02:18 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:02:18.674122 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:18.642347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:38881)
0925 15:02:18.651069 (+ 8722us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:18.651112 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:18.653223 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:18.654790 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:18.654804 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:18.655619 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:18.656402 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:18.656431 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:18.658828 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:18.658836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:18.660237 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:18.660249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:18.661769 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:18.661773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:18.661954 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:18.662758 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:18.662781 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:18.665180 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:18.670075 (+ 4895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:18.670082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:18.670094 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:18.670508 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:18.671067 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:18.671070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:18.671072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:18.671244 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:18.672252 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:18.672258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:18.672549 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:02:18.672787 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 15:02:18.673157 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8470,"thread_start_us":8295,"threads_started":1}
I20260925 15:02:18.674593 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:18.642004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:38881 (local address 127.4.243.254:33587)
0925 15:02:18.649241 (+ 7237us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:18.649253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:18.651260 (+ 2007us) server_negotiation.cc:407] Connection header received
0925 15:02:18.653889 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:18.653902 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:18.654240 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:18.654531 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:18.656660 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:18.658642 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:18.660386 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:18.661554 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:18.661743 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:18.665461 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:18.665474 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:18.665496 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:18.665550 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:18.669898 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:18.670692 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:18.670703 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:18.670714 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:18.670940 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:18.671370 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:18.671378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:18.671381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:18.672083 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:18.672294 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:18.673070 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 15:02:18.673527 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7029,"thread_start_us":6887,"threads_started":1}
I20260925 15:02:18.677263 21293 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33587
I20260925 15:02:18.677568 21293 heartbeater.cc:464] Registering TS with master...
I20260925 15:02:18.678305 21293 heartbeater.cc:511] Master 127.4.243.254:33587 requested a full tablet report, sending...
I20260925 15:02:18.680514 21107 ts_manager.cc:194] Registered new tserver with Master: 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003)
I20260925 15:02:18.681911 21107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:38881
W20260925 15:02:18.968251 21297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:18.968519 21297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:18.968660 21297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:18.975654 21297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:18.975752 21297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:18.975955 21297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:02:18.985450 21297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:42015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=35943
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:18.986793 21297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:18.988077 21297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:19.000525 21302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:19.000440 21303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:19.000484 21305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:19.001395 21297 server_base.cc:1034] running on GCE node
I20260925 15:02:19.001957 21297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:19.002882 21297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:19.004081 21297 hybrid_clock.cc:648] HybridClock initialized: now 1790348539004047 us; error 30 us; skew 500 ppm
I20260925 15:02:19.004355 21297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:19 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:19.007397 21297 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:02:19.008744 21297 webserver.cc:460] Webserver started at http://127.4.243.194:35943/ using document root <none> and password file <none>
I20260925 15:02:19.009341 21297 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:19.009443 21297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:19.014222 21297 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:02:19.016726 21312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:19.017968 21297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:02:19.018138 21297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "6274738a93c54ace9f752d91c7120d74"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "b536761c98d6969bf30b94ff7b4c6f14"
server_key_iv: "8cb339c02964c2005111f36e6abe37f6"
server_key_version: "encryptionkey@0"
I20260925 15:02:19.019097 21297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:19.038532 21297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:19.042404 21297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:19.042711 21297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:19.043635 21297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:19.044785 21297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:02:19.044876 21297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:19.044953 21297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:19.045006 21297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:19.077381 21297 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:42015
I20260925 15:02:19.077462 21424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:42015 every 8 connection(s)
I20260925 15:02:19.078822 21297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:02:19.082538 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21297
I20260925 15:02:19.083282 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:42363
--local_ip_for_outbound_sockets=127.4.243.195
--tserver_master_addrs=127.4.243.254:33587
--webserver_port=35467
--webserver_interface=127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:02:19 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:02:19.101150 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.081226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:37545 (local address 127.4.243.254:33587)
0925 15:02:19.081437 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.081444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.082652 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 15:02:19.085365 (+ 2713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.085369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.085438 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.085520 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:19.087539 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.088212 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.090547 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.090977 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.091060 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.094590 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:19.094598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:19.094630 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:19.094670 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:19.098118 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.098810 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.098817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.098821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.098958 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.099349 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.099357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.099363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.099720 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:19.099868 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.100481 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.100728 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:02:19.101761 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.081710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:37545)
0925 15:02:19.082481 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:19.082531 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:19.085103 (+ 2572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:19.085827 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:19.085840 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:19.086587 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:19.087367 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.087394 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.088368 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.088376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.090421 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.090436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.091184 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.091192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.091478 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.092168 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:19.092192 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:19.094360 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:19.098245 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:19.098251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:19.098261 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:19.098688 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:19.099058 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:19.099061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:19.099064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:19.099223 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:19.099833 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:19.099838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:19.100123 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 15:02:19.100330 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 15:02:19.100673 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":260,"threads_started":1}
I20260925 15:02:19.103381 21425 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33587
I20260925 15:02:19.103724 21425 heartbeater.cc:464] Registering TS with master...
I20260925 15:02:19.104698 21425 heartbeater.cc:511] Master 127.4.243.254:33587 requested a full tablet report, sending...
I20260925 15:02:19.106429 21107 ts_manager.cc:194] Registered new tserver with Master: 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015)
I20260925 15:02:19.107470 21107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:37545
W20260925 15:02:19.407007 21428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:19.407279 21428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:19.407437 21428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:19.414402 21428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:19.414496 21428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:19.414692 21428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:02:19.424249 21428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:42363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=35467
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:19.425558 21428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:19.426946 21428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:19.439739 21433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:19.439713 21434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:19.439814 21436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:19.440341 21428 server_base.cc:1034] running on GCE node
I20260925 15:02:19.440932 21428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:19.441782 21428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:19.442998 21428 hybrid_clock.cc:648] HybridClock initialized: now 1790348539442944 us; error 42 us; skew 500 ppm
I20260925 15:02:19.443302 21428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:19 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:19.446377 21428 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:02:19.447789 21428 webserver.cc:460] Webserver started at http://127.4.243.195:35467/ using document root <none> and password file <none>
I20260925 15:02:19.448501 21428 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:19.448655 21428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:19.453644 21428 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:02:19.456223 21443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:19.457463 21428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:02:19.457629 21428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "2b10ea8147bf48ffbea9d544a945c8e9"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "853fe3e46dafd1f941a722e53da1b23c"
server_key_iv: "da9a5adecab05a31474747d14eea6815"
server_key_version: "encryptionkey@0"
I20260925 15:02:19.458626 21428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:19.485792 21428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:19.490031 21428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:19.490319 21428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:19.491266 21428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:19.492504 21428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:02:19.492619 21428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:19.492740 21428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:19.492828 21428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:19.526153 21428 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42363
I20260925 15:02:19.526242 21555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42363 every 8 connection(s)
I20260925 15:02:19.527707 21428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:02:19.529551 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21428
Sep 25 15:02:19 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:02:19.544291 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.530546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.4.243.254:33587)
0925 15:02:19.531029 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.531037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.531363 (+ 326us) server_negotiation.cc:407] Connection header received
0925 15:02:19.531567 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.531579 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.531648 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.531764 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:19.533279 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.534011 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.535417 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.535933 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.536086 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.538847 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:19.538852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:19.538855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:19.538884 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:19.541373 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.542067 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.542072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.542074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.542216 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.542633 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.542639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.542642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.542954 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:19.543227 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.543596 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.543867 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":374,"thread_start_us":273,"threads_started":1}
I20260925 15:02:19.547998 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.530406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:41049 (local address 127.4.243.254:33587)
0925 15:02:19.530619 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.530625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.532120 (+ 1495us) server_negotiation.cc:407] Connection header received
0925 15:02:19.533893 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.533898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.533974 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.534094 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:19.535631 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.536409 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.537923 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.538318 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.538447 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.541899 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:19.541903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:19.541906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:19.541934 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:19.544840 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.545716 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.545721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.545723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.545880 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.546410 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.546413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.546415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.546737 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:19.546895 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.547607 (+ 712us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.547759 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:02:19.548969 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.531135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:41049)
0925 15:02:19.531966 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:19.532010 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:19.533641 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:19.534329 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:19.534337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:19.534881 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:19.535465 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.535481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.536562 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.536572 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.537755 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.537766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.538536 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.538542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.538816 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.539489 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:19.539522 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:19.541696 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:19.544973 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:19.544981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:19.544998 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:19.545525 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:19.546008 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:19.546014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:19.546019 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:19.546286 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:19.546898 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:19.546906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:19.547285 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 15:02:19.547506 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 15:02:19.547916 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":335,"threads_started":1}
I20260925 15:02:19.550192 21556 heartbeater.cc:347] Connected to a master server at 127.4.243.254:33587
I20260925 15:02:19.550480 21556 heartbeater.cc:464] Registering TS with master...
I20260925 15:02:19.551163 21556 heartbeater.cc:511] Master 127.4.243.254:33587 requested a full tablet report, sending...
I20260925 15:02:19.552472 21107 ts_manager.cc:194] Registered new tserver with Master: 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363)
I20260925 15:02:19.553498 21107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:41049
I20260925 15:02:19.559198 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:02:19.565721 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.560130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.4.243.254:33587)
0925 15:02:19.560336 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.560343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.560660 (+ 317us) server_negotiation.cc:407] Connection header received
0925 15:02:19.560892 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.560896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.561014 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.561156 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:19.561561 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.562317 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.563869 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.564335 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.564461 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.564916 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:19.565091 (+ 175us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:02:19.565437 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45130: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 15:02:19.565860 21295 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45130: token expired
W20260925 15:02:19.566300 21030 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.243.254:33587)
I20260925 15:02:19.578698 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.566780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.4.243.254:33587)
0925 15:02:19.566929 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.566935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.566977 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:02:19.567143 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.567148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.567234 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.567331 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:19.568551 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.569248 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.570588 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.571007 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.571134 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.573626 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:19.573632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:19.573636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:19.573674 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:19.576064 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.576764 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.576769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.576786 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.576896 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.577282 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.577290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.577294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.577577 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:19.577835 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.578117 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.578323 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:02:19.582031 21107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:02:19.589295 21107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:02:19.627624 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.601174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.4.243.194:42015)
0925 15:02:19.618426 (+ 17252us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.618432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.618458 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:02:19.618532 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.618537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.618850 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.619047 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:19.620012 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.621590 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.624867 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.626301 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.626446 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.626661 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.627258 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.627396 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17133,"thread_start_us":17071,"threads_started":1}
I20260925 15:02:19.627717 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.600899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42363 (local address 127.0.0.1:33274)
0925 15:02:19.619480 (+ 18581us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:19.619497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:19.619619 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:19.620492 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:19.620495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:19.620513 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:19.620760 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.620767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.622574 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.622581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.624979 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.624988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.626637 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.626641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.626847 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.626949 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 15:02:19.627080 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:02:19.627308 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18432,"mutex_wait_us":45,"thread_start_us":16806,"threads_started":1}
I20260925 15:02:19.627791 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.600649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.4.243.193:37003)
0925 15:02:19.619062 (+ 18413us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.619069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.619095 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:02:19.619183 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.619189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.619414 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.619602 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:19.620391 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.621783 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.624507 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.626006 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.626149 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.626362 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.627298 (+ 936us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.627465 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18250,"thread_start_us":18135,"threads_started":1}
I20260925 15:02:19.628306 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.600352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42015 (local address 127.0.0.1:56400)
0925 15:02:19.617977 (+ 17625us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:19.617994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:19.618083 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:19.619316 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:19.619322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:19.619354 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:19.619788 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.619803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.621742 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.621748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.624712 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.624726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.626821 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.626828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.626985 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.627030 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:02:19.627113 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:02:19.627946 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17494,"mutex_wait_us":799,"thread_start_us":15995,"threads_started":1}
I20260925 15:02:19.628468 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.600223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57674)
0925 15:02:19.615834 (+ 15611us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:19.615893 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:19.616115 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:19.619824 (+ 3709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:19.619830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:19.619860 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:19.620211 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.620222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.621928 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.621951 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.624351 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.624363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.626624 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.626631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.626899 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.626962 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 15:02:19.627080 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 15:02:19.628080 (+ 1000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15303,"mutex_wait_us":47,"thread_start_us":18938,"threads_started":1}
I20260925 15:02:19.628681 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.601598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.4.243.195:42363)
0925 15:02:19.617495 (+ 15897us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.617507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.619761 (+ 2254us) server_negotiation.cc:407] Connection header received
0925 15:02:19.619858 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.619865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.620068 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.620240 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:19.620939 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.622136 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.625120 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.626060 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.626211 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.626457 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.628171 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.628341 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15700,"thread_start_us":15522,"threads_started":1}
I20260925 15:02:19.630295 21228 tablet_service.cc:1467] Processing CreateTablet for tablet f98339a3aad449838b4e86aa8fd4b6e9 (DEFAULT_TABLE table=test-table [id=b6283d7e0faa427d9d8acaa9e63a5bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:02:19.631585 21228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98339a3aad449838b4e86aa8fd4b6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:19.632184 21360 tablet_service.cc:1467] Processing CreateTablet for tablet f98339a3aad449838b4e86aa8fd4b6e9 (DEFAULT_TABLE table=test-table [id=b6283d7e0faa427d9d8acaa9e63a5bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:02:19.632174 21491 tablet_service.cc:1467] Processing CreateTablet for tablet f98339a3aad449838b4e86aa8fd4b6e9 (DEFAULT_TABLE table=test-table [id=b6283d7e0faa427d9d8acaa9e63a5bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:02:19.633455 21360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98339a3aad449838b4e86aa8fd4b6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:19.633450 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98339a3aad449838b4e86aa8fd4b6e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:19.649442 21572 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Bootstrap starting.
I20260925 15:02:19.652201 21574 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Bootstrap starting.
I20260925 15:02:19.653956 21572 tablet_bootstrap.cc:654] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:19.655149 21572 log.cc:824] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:19.655309 21573 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Bootstrap starting.
I20260925 15:02:19.655401 21574 tablet_bootstrap.cc:654] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:19.656801 21574 log.cc:824] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:19.658249 21572 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: No bootstrap required, opened a new log
I20260925 15:02:19.658485 21572 ts_tablet_manager.cc:1397] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:02:19.658504 21573 tablet_bootstrap.cc:654] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:19.659391 21573 log.cc:824] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:19.659715 21574 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: No bootstrap required, opened a new log
I20260925 15:02:19.659966 21574 ts_tablet_manager.cc:1397] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 15:02:19.669377 21572 raft_consensus.cc:348] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.669821 21572 raft_consensus.cc:374] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:19.669916 21572 raft_consensus.cc:729] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfb15597ec14588817ce424a9412ace, State: Initialized, Role: FOLLOWER
I20260925 15:02:19.670333 21574 raft_consensus.cc:348] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.670806 21574 raft_consensus.cc:374] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:19.670625 21572 consensus_queue.cc:260] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.670900 21574 raft_consensus.cc:729] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b10ea8147bf48ffbea9d544a945c8e9, State: Initialized, Role: FOLLOWER
I20260925 15:02:19.671551 21574 consensus_queue.cc:260] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.672521 21573 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: No bootstrap required, opened a new log
I20260925 15:02:19.672787 21573 ts_tablet_manager.cc:1397] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260925 15:02:19.677811 21293 heartbeater.cc:503] Master 127.4.243.254:33587 was elected leader, sending a full tablet report...
I20260925 15:02:19.679067 21572 ts_tablet_manager.cc:1428] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Time spent starting tablet: real 0.020s user 0.013s sys 0.009s
I20260925 15:02:19.680465 21556 heartbeater.cc:503] Master 127.4.243.254:33587 was elected leader, sending a full tablet report...
I20260925 15:02:19.680533 21573 raft_consensus.cc:348] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.680915 21573 raft_consensus.cc:374] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:19.681007 21573 raft_consensus.cc:729] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6274738a93c54ace9f752d91c7120d74, State: Initialized, Role: FOLLOWER
I20260925 15:02:19.681578 21573 consensus_queue.cc:260] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.682024 21574 ts_tablet_manager.cc:1428] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Time spent starting tablet: real 0.022s user 0.020s sys 0.000s
I20260925 15:02:19.682839 21425 heartbeater.cc:503] Master 127.4.243.254:33587 was elected leader, sending a full tablet report...
I20260925 15:02:19.683558 21573 ts_tablet_manager.cc:1428] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260925 15:02:19.779407 21557 tablet.cc:2369] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:02:19.805075 21579 raft_consensus.cc:482] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:02:19.805405 21579 raft_consensus.cc:504] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.807454 21579 leader_election.cc:288] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003), 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015)
I20260925 15:02:19.815225 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.807868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.4.243.195:46465)
0925 15:02:19.808090 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:19.808110 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:19.808267 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:19.808909 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:19.808913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:19.808937 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:19.809138 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.809154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.810237 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.810245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.812749 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.812768 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.814386 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.814392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.814579 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.814674 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 15:02:19.814784 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:02:19.814916 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:02:19.815269 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.807945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42015 (local address 127.4.243.195:60081)
0925 15:02:19.808606 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:19.808629 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:19.808770 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:19.809289 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:19.809296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:19.809325 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:19.809808 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.809822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.810974 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.810980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.812385 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:19.812396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.814246 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.814255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:19.814500 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.814553 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:02:19.814662 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:02:19.814838 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":421,"threads_started":1}
I20260925 15:02:19.815887 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.808411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:46465 (local address 127.4.243.193:37003)
0925 15:02:19.808608 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.808614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.808632 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:02:19.808706 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.808710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.808787 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.808886 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:19.809279 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.810074 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.812916 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.814255 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.814401 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.814467 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.815510 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.815645 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:02:19.816442 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.808616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:60081 (local address 127.4.243.194:42015)
0925 15:02:19.808835 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.808841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.808859 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:02:19.808935 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.808943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.809028 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.809117 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:19.809997 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.810855 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.812684 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.814025 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.814195 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.814270 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.816122 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.816221 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:02:19.816946 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98339a3aad449838b4e86aa8fd4b6e9" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfb15597ec14588817ce424a9412ace" is_pre_election: true
I20260925 15:02:19.817359 21380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98339a3aad449838b4e86aa8fd4b6e9" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6274738a93c54ace9f752d91c7120d74" is_pre_election: true
I20260925 15:02:19.817463 21248 raft_consensus.cc:2454] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 0.
I20260925 15:02:19.817847 21380 raft_consensus.cc:2454] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 0.
I20260925 15:02:19.818195 21445 leader_election.cc:302] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bfb15597ec14588817ce424a9412ace, 2b10ea8147bf48ffbea9d544a945c8e9; no voters:
I20260925 15:02:19.818583 21579 raft_consensus.cc:2781] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:02:19.818758 21579 raft_consensus.cc:482] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:02:19.818867 21579 raft_consensus.cc:3037] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:19.821324 21579 raft_consensus.cc:504] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:19.821981 21579 leader_election.cc:288] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 election: Requested vote from peers 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003), 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015)
I20260925 15:02:19.822388 21248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98339a3aad449838b4e86aa8fd4b6e9" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfb15597ec14588817ce424a9412ace"
I20260925 15:02:19.822677 21380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98339a3aad449838b4e86aa8fd4b6e9" candidate_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6274738a93c54ace9f752d91c7120d74"
I20260925 15:02:19.822710 21248 raft_consensus.cc:3037] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:19.822894 21380 raft_consensus.cc:3037] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:19.826349 21380 raft_consensus.cc:2454] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 1.
I20260925 15:02:19.826346 21248 raft_consensus.cc:2454] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b10ea8147bf48ffbea9d544a945c8e9 in term 1.
I20260925 15:02:19.826834 21446 leader_election.cc:302] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74; no voters:
I20260925 15:02:19.827231 21579 raft_consensus.cc:2781] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:19.827633 21579 raft_consensus.cc:686] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 2b10ea8147bf48ffbea9d544a945c8e9, State: Running, Role: LEADER
I20260925 15:02:19.828186 21579 consensus_queue.cc:237] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
W20260925 15:02:19.830860 21426 tablet.cc:2369] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:02:19.833218 21106 catalog_manager.cc:5495] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195). New cstate: current_term: 1 leader_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } health_report { overall_health: HEALTHY } } }
I20260925 15:02:19.876849 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.871884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.4.243.195:42363)
0925 15:02:19.872098 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.872103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.872124 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:02:19.872209 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.872213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.872285 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.872430 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:19.872806 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.873495 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.874819 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.875205 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.875336 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.875787 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:19.876283 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:19.876436 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.876492 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.876569 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:02:19.884644 21248 raft_consensus.cc:1261] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Refusing update from remote peer 2b10ea8147bf48ffbea9d544a945c8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:19.884646 21380 raft_consensus.cc:1261] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Refusing update from remote peer 2b10ea8147bf48ffbea9d544a945c8e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:19.885576 21582 consensus_queue.cc:1035] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:02:19.885941 21579 consensus_queue.cc:1035] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:02:19.891084 21294 tablet.cc:2369] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:02:19.898896 21585 mvcc.cc:205] Tried to move back new op lower bound from 7333267619357483008 to 7333267619140485120. Current Snapshot: MvccSnapshot[applied={T|T < 7333267619357483008}]
I20260925 15:02:19.913758 21107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:02:19.914062 21107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:02:19.917580 21107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Sending DeleteTablet for 3 replicas of tablet f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:19.918763 21491 tablet_service.cc:1514] Processing DeleteTablet for tablet f98339a3aad449838b4e86aa8fd4b6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:02:19 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:33274
I20260925 15:02:19.919993 21360 tablet_service.cc:1514] Processing DeleteTablet for tablet f98339a3aad449838b4e86aa8fd4b6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:02:19 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:56400
I20260925 15:02:19.920109 21228 tablet_service.cc:1514] Processing DeleteTablet for tablet f98339a3aad449838b4e86aa8fd4b6e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:02:19 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:57674
I20260925 15:02:19.923002 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 21144
I20260925 15:02:19.924352 21595 tablet_replica.cc:330] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: stopping tablet replica
I20260925 15:02:19.926959 21595 raft_consensus.cc:2229] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:02:19.927484 21595 raft_consensus.cc:2258] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:02:19.928226 21594 tablet_replica.cc:330] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: stopping tablet replica
I20260925 15:02:19.928579 21594 raft_consensus.cc:2229] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:02:19.929596 21594 raft_consensus.cc:2258] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:02:19.935496 21595 ts_tablet_manager.cc:1905] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:02:19.938794 21594 ts_tablet_manager.cc:1905] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:02:19.946167 21596 tablet_replica.cc:330] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: stopping tablet replica
I20260925 15:02:19.947614 21596 raft_consensus.cc:2229] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:02:19.948218 21596 raft_consensus.cc:2258] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:02:19.947175 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:19.925845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.4.243.193:37003)
0925 15:02:19.926441 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:19.926447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:19.926467 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:02:19.926534 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:19.926539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:19.926639 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:19.926742 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:19.929560 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.931445 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.933958 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:19.934873 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:19.935043 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:19.938213 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:19.938233 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:19.938256 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:19.938313 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:19.943451 (+ 5138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.944467 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.944481 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.944487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.944717 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:19.945165 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:19.945170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:19.945176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:19.945650 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:19.945991 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:19.946476 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 15:02:19.946796 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477}
I20260925 15:02:19.949501 21288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:02:19.950155 21595 ts_tablet_manager.cc:1918] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:02:19.950335 21595 log.cc:1196] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:19.950249 21594 ts_tablet_manager.cc:1918] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:02:19.950845 21594 log.cc:1196] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:19.950889 21595 ts_tablet_manager.cc:1939] T f98339a3aad449838b4e86aa8fd4b6e9 P 6274738a93c54ace9f752d91c7120d74: Deleting consensus metadata
I20260925 15:02:19.951598 21594 ts_tablet_manager.cc:1939] T f98339a3aad449838b4e86aa8fd4b6e9 P 2b10ea8147bf48ffbea9d544a945c8e9: Deleting consensus metadata
I20260925 15:02:19.952857 21093 catalog_manager.cc:4841] TS 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015): tablet f98339a3aad449838b4e86aa8fd4b6e9 (table test-table [id=b6283d7e0faa427d9d8acaa9e63a5bb7]) successfully deleted
I20260925 15:02:19.953343 21094 catalog_manager.cc:4841] TS 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363): tablet f98339a3aad449838b4e86aa8fd4b6e9 (table test-table [id=b6283d7e0faa427d9d8acaa9e63a5bb7]) successfully deleted
I20260925 15:02:20.043612 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 21144
I20260925 15:02:20.060900 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 21297
W20260925 15:02:20.060945 21092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:37003 (error 108)
I20260925 15:02:20.063038 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.062466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57680)
0925 15:02:20.062750 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.062863 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260925 15:02:20.063285 21092 catalog_manager.cc:4584] TS 1bfb15597ec14588817ce424a9412ace (127.4.243.193:37003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f98339a3aad449838b4e86aa8fd4b6e9: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
I20260925 15:02:20.077098 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.061995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.4.243.194:42015)
0925 15:02:20.062203 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:20.062208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:20.062229 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:02:20.062456 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:20.062462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:20.062564 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:20.062670 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:20.064284 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.065448 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.066990 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.067632 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.067770 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:20.070184 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:20.070203 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:20.070222 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:20.070261 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:20.074064 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.074798 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.074807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.074811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.074951 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.075301 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.075306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.075310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.075752 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:20.076065 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:20.076206 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 15:02:20.076419 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:02:20.077802 21420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:02:20.117030 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.116487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57690)
0925 15:02:20.116722 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.116843 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:02:20.118145 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.117758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57704)
0925 15:02:20.117911 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.118008 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:02:20.167384 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 21297
I20260925 15:02:20.178864 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.178239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57710)
0925 15:02:20.178512 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.178660 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:02:20.180094 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.179640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57714)
0925 15:02:20.179838 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.179934 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:02:20.185856 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 21428
I20260925 15:02:20.200189 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.186772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.4.243.195:42363)
0925 15:02:20.186953 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:20.186960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:20.186985 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:02:20.187060 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:20.187065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:20.187166 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:20.187286 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:20.188605 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.189352 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.190990 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.191432 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.191571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:20.194189 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:20.194200 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:20.194215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:20.194295 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:20.197425 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.198100 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.198107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.198113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.198266 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.198642 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.198647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.198653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.198999 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:20.199342 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:20.199577 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 15:02:20.199804 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:02:20.201114 21551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:02:20.260876 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.260302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57724)
0925 15:02:20.260568 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.260680 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 15:02:20.261948 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.261526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57730)
0925 15:02:20.261729 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.261815 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 15:02:20.325037 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 21428
I20260925 15:02:20.343809 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 21074
I20260925 15:02:20.345045 21137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:02:20.459544 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.459023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:37003 (local address 127.0.0.1:57732)
0925 15:02:20.459266 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:20.459367 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:37003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:02:20.459574 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 21074
I20260925 15:02:40.481285 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33587
--webserver_interface=127.4.243.254
--webserver_port=37439
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:40.804682 21602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:40.804942 21602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:40.805127 21602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:40.812043 21602 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:02:40.812155 21602 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:02:40.812219 21602 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:02:40.812343 21602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:40.812431 21602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:40.812539 21602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:40.812616 21602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:40.812716 21602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:40.821942 21602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:33587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=37439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:40.823138 21602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:40.824338 21602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:40.835640 21608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:40.835714 21610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:40.835750 21607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:40.836423 21602 server_base.cc:1034] running on GCE node
I20260925 15:02:40.837249 21602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:40.838527 21602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:40.839790 21602 hybrid_clock.cc:648] HybridClock initialized: now 1790348560839726 us; error 37 us; skew 500 ppm
I20260925 15:02:40.840121 21602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:40 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348560, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:40.843291 21602 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:02:40.844781 21602 webserver.cc:460] Webserver started at http://127.4.243.254:37439/ using document root <none> and password file <none>
I20260925 15:02:40.845436 21602 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:40.845521 21602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:40.850728 21602 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:02:40.853358 21617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:40.854507 21602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:02:40.854729 21602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "1b479d2b57a04275afa44e011012af33"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "65db8d5027988d3a0c7f847719aef87e"
server_key_iv: "8af2dd513c34d2150c0b11cfdd3c6809"
server_key_version: "encryptionkey@0"
I20260925 15:02:40.855598 21602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:40.887295 21602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:40.891162 21602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:40.891443 21602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:40.909673 21602 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:33587
I20260925 15:02:40.909720 21668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:33587 every 8 connection(s)
I20260925 15:02:40.911221 21602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:40.913921 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21602
I20260925 15:02:40.915086 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:37003
--local_ip_for_outbound_sockets=127.4.243.193
--tserver_master_addrs=127.4.243.254:33587
--webserver_port=37651
--webserver_interface=127.4.243.193
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:02:40.919243 21669 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:02:40.923266 21669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Bootstrap starting.
I20260925 15:02:40.952618 21669 log.cc:824] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:40.972860 21669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:02:40.973547 21669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Bootstrap complete.
I20260925 15:02:40.988138 21669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:40.989387 21669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b479d2b57a04275afa44e011012af33, State: Initialized, Role: FOLLOWER
I20260925 15:02:40.990139 21669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:40.990414 21669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:40.990531 21669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:40.990737 21669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:02:40.996188 21669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:40.996755 21669 leader_election.cc:302] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b479d2b57a04275afa44e011012af33; no voters:
I20260925 15:02:40.997447 21669 leader_election.cc:288] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:02:40.997614 21673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:02:40.998975 21673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [term 3 LEADER]: Becoming Leader. State: Replica: 1b479d2b57a04275afa44e011012af33, State: Running, Role: LEADER
I20260925 15:02:40.999529 21669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:40.999496 21673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } }
I20260925 15:02:41.001703 21675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b479d2b57a04275afa44e011012af33. Latest consensus state: current_term: 3 leader_uuid: "1b479d2b57a04275afa44e011012af33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } } }
I20260925 15:02:41.002071 21675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:41.002571 21674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1b479d2b57a04275afa44e011012af33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b479d2b57a04275afa44e011012af33" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 33587 } } }
I20260925 15:02:41.002956 21674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:41.004318 21679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:41.012703 21679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:41.014099 21679 catalog_manager.cc:1259] Loaded cluster ID: ae6f3436a31844fd9273e3fba9a4dc5d
I20260925 15:02:41.014196 21679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:41.018172 21679 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:41.019719 21679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Loaded TSK: 0
I20260925 15:02:41.019878 21679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b479d2b57a04275afa44e011012af33: Loaded TSK: 1
F20260925 15:02:41.020910 21679 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff34d0539ad google::LogMessage::Fail() at ??:0
@ 0x7ff34d05688a google::LogMessage::SendToLog() at ??:0
@ 0x7ff34d0534de google::LogMessage::Flush() at ??:0
@ 0x7ff34d054a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff350db77d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff350da673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff362a821c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff362a81783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff362b0d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff362b0d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff362b575e9 std::function<>::operator()() at ??:0
@ 0x7ff362a85cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff362a83c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff362b0b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff362b0b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff34fc9f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff34fca291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff34fca2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff34fc7deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff352f576db start_thread at ??:0
@ 0x7ff34beed61f clone at ??:0
W20260925 15:02:41.310400 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:41.310781 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:41.310981 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:41.318173 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:41.318320 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:41.318507 21671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:02:41.327967 21671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:37003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=37651
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:41.329182 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:41.330493 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:41.343454 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:41.343489 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:41.343508 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:41.344220 21671 server_base.cc:1034] running on GCE node
I20260925 15:02:41.344790 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:41.345763 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:41.346978 21671 hybrid_clock.cc:648] HybridClock initialized: now 1790348561346932 us; error 63 us; skew 500 ppm
I20260925 15:02:41.347281 21671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:41 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348561, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:41.350409 21671 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:02:41.351933 21671 webserver.cc:460] Webserver started at http://127.4.243.193:37651/ using document root <none> and password file <none>
I20260925 15:02:41.352557 21671 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:41.352654 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:41.357800 21671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:02:41.360564 21726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:41.361807 21671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:02:41.361929 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "1bfb15597ec14588817ce424a9412ace"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-qxh0"
server_key: "9c3354967ce2a3a0bf2004e3c7a8132e"
server_key_iv: "e764ac81ef7adee7c98d7c7655ede8ef"
server_key_version: "encryptionkey@0"
I20260925 15:02:41.362913 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:41.379494 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:41.383415 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:41.383612 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:41.384490 21671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:41.386062 21733 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 15:02:41.391551 21671 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:02:41.391734 21671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 15:02:41.391887 21671 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:02:41.394991 21671 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:02:41.395097 21671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260925 15:02:41.395411 21733 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Bootstrap starting.
I20260925 15:02:41.403704 21733 log.cc:824] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:41.417050 21733 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:02:41.418010 21733 tablet_bootstrap.cc:492] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Bootstrap complete.
I20260925 15:02:41.418471 21733 ts_tablet_manager.cc:1397] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20260925 15:02:41.429589 21733 raft_consensus.cc:348] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:41.430730 21733 raft_consensus.cc:729] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfb15597ec14588817ce424a9412ace, State: Initialized, Role: FOLLOWER
I20260925 15:02:41.431392 21733 consensus_queue.cc:260] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:41.433027 21733 ts_tablet_manager.cc:1428] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 15:02:41.440259 21671 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:37003
I20260925 15:02:41.440336 21841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:37003 every 8 connection(s)
I20260925 15:02:41.441479 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:02:41.445415 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21671
I20260925 15:02:41.446030 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:41.444195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:59003)
0925 15:02:41.444904 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:41.445144 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
I20260925 15:02:41.446038 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:42015
--local_ip_for_outbound_sockets=127.4.243.194
--tserver_master_addrs=127.4.243.254:33587
--webserver_port=35943
--webserver_interface=127.4.243.194
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:41.446712 21842 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:33587 (0 consecutive failures): Network error: Failed to ping master at 127.4.243.254:33587: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
I20260925 15:02:41.447642 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:41.447208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:60099)
0925 15:02:41.447394 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:41.447487 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:02:41.448642 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:41.448216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:42791)
0925 15:02:41.448393 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:41.448490 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:02:41.448887 21842 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:02:41.764039 21845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:41.764324 21845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:41.764480 21845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:41.771587 21845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:41.771724 21845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:41.771898 21845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:02:41.780946 21845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:42015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=35943
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:41.782125 21845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:41.783433 21845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:41.796492 21850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:41.796522 21853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:41.796587 21851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:41.797385 21845 server_base.cc:1034] running on GCE node
I20260925 15:02:41.797932 21845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:41.798888 21845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:41.800081 21845 hybrid_clock.cc:648] HybridClock initialized: now 1790348561800036 us; error 43 us; skew 500 ppm
I20260925 15:02:41.800359 21845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:41 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348561, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:41.803282 21845 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:02:41.804647 21845 webserver.cc:460] Webserver started at http://127.4.243.194:35943/ using document root <none> and password file <none>
I20260925 15:02:41.805313 21845 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:41.805444 21845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:41.810864 21845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:02:41.813349 21860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:41.814492 21845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:02:41.814718 21845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "6274738a93c54ace9f752d91c7120d74"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "b536761c98d6969bf30b94ff7b4c6f14"
server_key_iv: "8cb339c02964c2005111f36e6abe37f6"
server_key_version: "encryptionkey@0"
I20260925 15:02:41.815610 21845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:41.843426 21845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:41.848407 21845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:41.848686 21845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:41.849699 21845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:41.851132 21845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:02:41.851227 21845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:41.851380 21845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:41.851469 21845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:41.884490 21845 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:42015
I20260925 15:02:41.884591 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:42015 every 8 connection(s)
I20260925 15:02:41.885993 21845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:02:41.888775 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21845
I20260925 15:02:41.889389 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:42363
--local_ip_for_outbound_sockets=127.4.243.195
--tserver_master_addrs=127.4.243.254:33587
--webserver_port=35467
--webserver_interface=127.4.243.195
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:02:41.891062 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:41.889020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:51927)
0925 15:02:41.889857 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:41.890151 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":346,"threads_started":1}
W20260925 15:02:41.891752 21973 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:33587 (0 consecutive failures): Network error: Failed to ping master at 127.4.243.254:33587: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
I20260925 15:02:41.892609 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:41.892316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:37437)
0925 15:02:41.892452 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:41.892515 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:02:41.893366 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:41.893039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:57607)
0925 15:02:41.893175 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:41.893247 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:02:41.893566 21973 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:02:42.214936 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:42.215197 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:42.215291 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:42.223016 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:42.223115 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:42.223227 21976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:02:42.233338 21976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:42363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=35467
--tserver_master_addrs=127.4.243.254:33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:42.234571 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:42.235942 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:42.248728 21984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:42.248802 21981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:42.248775 21982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:42.249768 21976 server_base.cc:1034] running on GCE node
I20260925 15:02:42.250325 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:42.251313 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:42.252535 21976 hybrid_clock.cc:648] HybridClock initialized: now 1790348562252500 us; error 47 us; skew 500 ppm
I20260925 15:02:42.252836 21976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:42 dist-test-slave-qxh0 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348562, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:42.255920 21976 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:02:42.257180 21976 webserver.cc:460] Webserver started at http://127.4.243.195:35467/ using document root <none> and password file <none>
I20260925 15:02:42.257802 21976 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:42.257938 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:42.263031 21976 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:02:42.265625 21991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:42.267021 21976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:02:42.267200 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "2b10ea8147bf48ffbea9d544a945c8e9"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-qxh0"
server_key: "853fe3e46dafd1f941a722e53da1b23c"
server_key_iv: "da9a5adecab05a31474747d14eea6815"
server_key_version: "encryptionkey@0"
I20260925 15:02:42.268232 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:42.284274 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:42.288169 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:42.288440 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:42.289376 21976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:42.290719 21976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:02:42.290829 21976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:42.290961 21976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:42.291049 21976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:42.323648 21976 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:42363
I20260925 15:02:42.323737 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:42363 every 8 connection(s)
I20260925 15:02:42.325074 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:02:42.325775 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 21976
I20260925 15:02:42.329949 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:42.328238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:39123)
0925 15:02:42.328930 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:42.329141 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":281,"threads_started":1}
W20260925 15:02:42.330641 22104 heartbeater.cc:644] Failed to heartbeat to 127.4.243.254:33587 (0 consecutive failures): Network error: Failed to ping master at 127.4.243.254:33587: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
I20260925 15:02:42.331406 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:42.331095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:51803)
0925 15:02:42.331253 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:42.331314 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:02:42.332150 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:42.331853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:55977)
0925 15:02:42.331977 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:42.332052 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:02:42.332360 22104 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:02:42.450112 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:42.449327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:33641)
0925 15:02:42.449789 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:42.449898 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":241,"threads_started":1}
I20260925 15:02:42.894835 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:42.894045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:42471)
0925 15:02:42.894457 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:42.894537 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":197,"threads_started":1}
I20260925 15:02:43.194401 22110 raft_consensus.cc:482] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:02:43.194744 22110 raft_consensus.cc:504] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:43.196395 22110 leader_election.cc:288] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015), 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363)
Sep 25 15:02:43 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348561, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
Sep 25 15:02:43 dist-test-slave-qxh0 krb5kdc[20523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348561, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:02:43.229799 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:43.197023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:55575 (local address 127.4.243.194:42015)
0925 15:02:43.209533 (+ 12510us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:43.209550 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:43.209962 (+ 412us) server_negotiation.cc:407] Connection header received
0925 15:02:43.212256 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:43.212266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:43.212736 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:43.213096 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:43.215505 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.216653 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.218391 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.219132 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.219343 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:43.223027 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:43.223033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:43.223044 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:43.223079 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:43.226123 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:43.227033 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:43.227045 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:43.227049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:43.227178 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:43.227584 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:43.227592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:43.227595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:43.228073 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:43.228282 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:43.229037 (+ 755us) server_negotiation.cc:299] Negotiation successful
0925 15:02:43.229383 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12296,"thread_start_us":12098,"threads_started":1}
I20260925 15:02:43.229808 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:43.196679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:42363 (local address 127.4.243.193:43559)
0925 15:02:43.209092 (+ 12413us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:43.209141 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:43.209774 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:43.213265 (+ 3491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:43.213277 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:43.214518 (+ 1241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:43.215310 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:43.215331 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.216843 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.216847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:43.218233 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:43.218245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.219227 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.219233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:43.219511 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:43.220176 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:43.220204 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:43.222491 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:43.226237 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:43.226247 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:43.226260 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:43.226908 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:43.227297 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:43.227303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:43.227306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:43.227494 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:43.228218 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:43.228227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:43.228557 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 15:02:43.228808 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 15:02:43.229320 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12253,"mutex_wait_us":140,"spinlock_wait_cycles":72320,"thread_start_us":12116,"threads_started":1}
I20260925 15:02:43.229831 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:43.196984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:43559 (local address 127.4.243.195:42363)
0925 15:02:43.208734 (+ 11750us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:43.208758 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:43.209375 (+ 617us) server_negotiation.cc:407] Connection header received
0925 15:02:43.212210 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:43.212225 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:43.212726 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:43.213096 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:43.215496 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.216642 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.218384 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.219045 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.219250 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:43.222786 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:43.222792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:43.222804 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:43.222836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:43.226103 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:43.227041 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:43.227047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:43.227052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:43.227188 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:43.227607 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:43.227618 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:43.227620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:43.228076 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:43.228292 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:43.229032 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 15:02:43.229384 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11517,"thread_start_us":11377,"threads_started":1}
I20260925 15:02:43.229914 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:43.196710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:42015 (local address 127.4.243.193:55575)
0925 15:02:43.209815 (+ 13105us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:43.209830 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:43.209945 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:43.213265 (+ 3320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:43.213277 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:43.214472 (+ 1195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:43.215313 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:43.215335 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.216840 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.216844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:43.218244 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:43.218255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:43.219373 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:43.219380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:43.219614 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:43.220345 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:43.220362 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:43.222782 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:43.226255 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:43.226262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:43.226269 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:43.226901 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:43.227289 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:43.227295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:43.227298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:43.227468 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:43.228220 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:43.228230 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:43.228557 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 15:02:43.228807 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 15:02:43.229479 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12981,"mutex_wait_us":424,"spinlock_wait_cycles":3712,"thread_start_us":12835,"threads_started":1}
W20260925 15:02:43.233462 21729 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
W20260925 15:02:43.234164 21730 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:43.234297 21730 leader_election.cc:302] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bfb15597ec14588817ce424a9412ace; no voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74
I20260925 15:02:43.234695 22110 raft_consensus.cc:2726] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:02:43.333683 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:43.332823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:54779)
0925 15:02:43.333256 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:43.333386 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":217,"threads_started":1}
I20260925 15:02:43.451436 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:43.451003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:35461)
0925 15:02:43.451208 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:43.451304 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:02:43.896215 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:43.895579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:43977)
0925 15:02:43.895958 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:43.896069 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":220,"threads_started":1}
I20260925 15:02:44.335091 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:44.334459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:60399)
0925 15:02:44.334888 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:44.334958 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":248,"threads_started":1}
I20260925 15:02:44.452910 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:44.452219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:36997)
0925 15:02:44.452676 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:44.452748 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":254,"threads_started":1}
I20260925 15:02:44.897742 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:44.897043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:33617)
0925 15:02:44.897500 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:44.897595 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":255,"threads_started":1}
I20260925 15:02:44.947161 22120 raft_consensus.cc:482] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:02:44.947386 22120 raft_consensus.cc:504] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:44.948028 22120 leader_election.cc:288] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015), 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363)
W20260925 15:02:44.949110 21729 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
W20260925 15:02:44.949296 21730 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:44.949429 21730 leader_election.cc:302] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bfb15597ec14588817ce424a9412ace; no voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74
I20260925 15:02:44.949672 22120 raft_consensus.cc:2726] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:02:45.336444 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:45.335798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:34797)
0925 15:02:45.336228 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:45.336300 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260925 15:02:45.454394 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:45.453687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:60569)
0925 15:02:45.454088 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:45.454180 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":201,"threads_started":1}
I20260925 15:02:45.899325 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:45.898688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:38847)
0925 15:02:45.899088 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:45.899161 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":200,"threads_started":1}
I20260925 15:02:46.337849 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:46.337208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:42721)
0925 15:02:46.337607 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:46.337676 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":197,"threads_started":1}
I20260925 15:02:46.456097 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:46.455383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:55465)
0925 15:02:46.455833 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:46.455923 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":216,"threads_started":1}
I20260925 15:02:46.599884 22126 raft_consensus.cc:482] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:02:46.600126 22126 raft_consensus.cc:504] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:46.600746 22126 leader_election.cc:288] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015), 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363)
W20260925 15:02:46.601653 21730 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
W20260925 15:02:46.601886 21729 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:46.602025 21729 leader_election.cc:302] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bfb15597ec14588817ce424a9412ace; no voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74
I20260925 15:02:46.602290 22126 raft_consensus.cc:2726] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:02:46.900803 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:46.900111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:39305)
0925 15:02:46.900540 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:46.900613 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":255,"threads_started":1}
I20260925 15:02:47.339495 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:47.338853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:41517)
0925 15:02:47.339256 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:47.339331 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":228,"threads_started":1}
I20260925 15:02:47.457609 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:47.456924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:38533)
0925 15:02:47.457359 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:47.457447 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260925 15:02:47.902275 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:47.901602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:35973)
0925 15:02:47.902020 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:47.902100 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":226,"threads_started":1}
I20260925 15:02:48.341195 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:48.340387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:48287)
0925 15:02:48.340897 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:48.340996 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":281,"threads_started":1}
I20260925 15:02:48.459378 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:48.458537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:37265)
0925 15:02:48.459096 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:48.459182 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":371,"threads_started":1}
I20260925 15:02:48.903823 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:48.903089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:49275)
0925 15:02:48.903557 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:48.903642 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":217,"threads_started":1}
I20260925 15:02:49.246760 22134 raft_consensus.cc:482] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:02:49.246950 22134 raft_consensus.cc:504] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:49.247671 22134 leader_election.cc:288] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015), 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363)
W20260925 15:02:49.248713 21729 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
W20260925 15:02:49.248980 21730 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:49.249141 21730 leader_election.cc:302] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bfb15597ec14588817ce424a9412ace; no voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74
I20260925 15:02:49.249399 22134 raft_consensus.cc:2726] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:02:49.342833 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:49.342002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:49621)
0925 15:02:49.342493 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:49.342574 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":221,"threads_started":1}
I20260925 15:02:49.461105 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:49.460241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:34879)
0925 15:02:49.460840 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:49.460931 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":306,"threads_started":1}
I20260925 15:02:49.905308 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:49.904674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:47467)
0925 15:02:49.905051 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:49.905136 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":190,"threads_started":1}
I20260925 15:02:50.344370 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:50.343699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:41575)
0925 15:02:50.344094 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:50.344169 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":204,"threads_started":1}
I20260925 15:02:50.462659 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:50.461925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:58499)
0925 15:02:50.462340 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:50.462418 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260925 15:02:50.907102 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:50.906350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:42531)
0925 15:02:50.906847 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:50.906942 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260925 15:02:51.345945 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:51.345247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:36113)
0925 15:02:51.345718 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:51.345796 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260925 15:02:51.464205 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:51.463536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:35357)
0925 15:02:51.463966 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:51.464041 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":250,"threads_started":1}
I20260925 15:02:51.908622 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:51.907938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:34263)
0925 15:02:51.908357 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:51.908444 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":227,"threads_started":1}
I20260925 15:02:52.347508 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:52.346832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:37181)
0925 15:02:52.347262 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:52.347341 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":230,"threads_started":1}
I20260925 15:02:52.465860 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:52.465114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:47213)
0925 15:02:52.465567 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:52.465688 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":267,"threads_started":1}
I20260925 15:02:52.910389 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:52.909595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:53945)
0925 15:02:52.910143 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:52.910226 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
I20260925 15:02:53.349180 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:53.348511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:41457)
0925 15:02:53.348947 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:53.349029 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":218,"threads_started":1}
I20260925 15:02:53.467417 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:53.466704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:45247)
0925 15:02:53.467157 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:53.467258 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":220,"threads_started":1}
I20260925 15:02:53.911926 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:53.911221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:44865)
0925 15:02:53.911683 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:53.911767 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":236,"threads_started":1}
I20260925 15:02:54.350811 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:54.349994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:39705)
0925 15:02:54.350447 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:54.350521 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
I20260925 15:02:54.468954 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:54.468221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:53791)
0925 15:02:54.468691 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:54.468780 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260925 15:02:54.913551 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:54.912880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:42345)
0925 15:02:54.913302 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:54.913396 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
I20260925 15:02:55.075922 22153 raft_consensus.cc:482] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:02:55.076128 22153 raft_consensus.cc:504] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb15597ec14588817ce424a9412ace" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 37003 } } peers { permanent_uuid: "6274738a93c54ace9f752d91c7120d74" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 42015 } } peers { permanent_uuid: "2b10ea8147bf48ffbea9d544a945c8e9" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 42363 } }
I20260925 15:02:55.076799 22153 leader_election.cc:288] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015), 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363)
W20260925 15:02:55.077746 21730 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b10ea8147bf48ffbea9d544a945c8e9 (127.4.243.195:42363): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
W20260925 15:02:55.077925 21729 leader_election.cc:341] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6274738a93c54ace9f752d91c7120d74 (127.4.243.194:42015): Not found: Tablet not found: f98339a3aad449838b4e86aa8fd4b6e9
I20260925 15:02:55.078039 21729 leader_election.cc:302] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bfb15597ec14588817ce424a9412ace; no voters: 2b10ea8147bf48ffbea9d544a945c8e9, 6274738a93c54ace9f752d91c7120d74
I20260925 15:02:55.078302 22153 raft_consensus.cc:2726] T f98339a3aad449838b4e86aa8fd4b6e9 P 1bfb15597ec14588817ce424a9412ace [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:02:55.352530 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:55.351753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:50123)
0925 15:02:55.352285 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:55.352375 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":290,"threads_started":1}
I20260925 15:02:55.470894 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:55.469922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:40877)
0925 15:02:55.470424 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:55.470550 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":251,"threads_started":1}
I20260925 15:02:55.915264 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:55.914521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:46083)
0925 15:02:55.914997 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:55.915075 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260925 15:02:56.354130 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:56.353367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:57875)
0925 15:02:56.353850 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:56.353937 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260925 15:02:56.472815 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:56.471922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.193:46219)
0925 15:02:56.472536 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:56.472614 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":358,"threads_started":1}
I20260925 15:02:56.917268 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:56.916507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.194:36463)
0925 15:02:56.917016 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:56.917101 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":284,"threads_started":1}
I20260925 15:02:57.333143 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 21671
I20260925 15:02:57.356036 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:57.355253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:33587 (local address 127.4.243.195:41381)
0925 15:02:57.355747 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:57.355887 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.254:33587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260925 15:02:57.356709 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:57.334087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.4.243.193:37003)
0925 15:02:57.334542 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:57.334548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:57.334735 (+ 187us) server_negotiation.cc:407] Connection header received
0925 15:02:57.334884 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:57.334891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:57.335112 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:57.335243 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:57.336590 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:57.337565 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:57.338851 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:57.339423 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:57.339574 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:57.351040 (+ 11466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:57.351053 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:57.351067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:57.351107 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:57.353636 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:57.354354 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:57.354362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:57.354364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:57.354545 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:57.355012 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:57.355017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:57.355020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:57.355392 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:57.355741 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:57.356112 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 15:02:57.356407 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":198,"threads_started":1}
I20260925 15:02:57.357925 21837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:02:57.447834 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 21671
I20260925 15:02:57.464983 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 21845
I20260925 15:02:57.477918 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:57.465746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.4.243.194:42015)
0925 15:02:57.466236 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:57.466240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:57.466256 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:02:57.466310 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:57.466314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:57.466373 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:57.466466 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:57.467777 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:57.468564 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:57.469766 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:57.470190 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:57.470295 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:57.472892 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:57.472897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:57.472900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:57.472932 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:57.475519 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:57.476137 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:57.476141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:57.476143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:57.476275 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:57.476586 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:57.476590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:57.476592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:57.476853 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:57.477063 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:57.477246 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:02:57.477411 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260925 15:02:57.478683 21968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:02:57.561421 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 21845
I20260925 15:02:57.580099 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 21976
I20260925 15:02:57.593822 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:57.580951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.4.243.195:42363)
0925 15:02:57.581525 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:57.581529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:57.581545 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:02:57.581599 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:57.581603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:57.581668 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:57.581783 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:57.583123 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:57.583941 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:57.585069 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:57.585613 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:57.585797 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:57.588506 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:57.588510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:57.588514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:57.588542 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:57.591360 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:57.591996 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:57.592002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:57.592005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:57.592136 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:57.592500 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:57.592505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:57.592508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:57.592752 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:57.593018 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:57.593280 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:02:57.593475 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":319,"threads_started":1}
I20260925 15:02:57.594471 22099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:02:57.676596 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 21976
2026-09-25T15:02:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64107 ms)
[----------] 30 tests from SecurityITest (293157 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:57 dist-test-slave-qxh0 krb5kdc[22169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:57 dist-test-slave-qxh0 krb5kdc[22169](info): set up 2 sockets
Sep 25 15:02:57 dist-test-slave-qxh0 krb5kdc[22169](info): commencing operation
krb5kdc: starting...
W20260925 15:02:58.774528 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:02:58 dist-test-slave-qxh0 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:02:59.008102 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38917
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:38917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:59.328456 22182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:59.328784 22182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:59.328951 22182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:59.335791 22182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:59.335940 22182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:59.336062 22182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:59.336164 22182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:59.336246 22182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:59.345357 22182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:38917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:59.346589 22182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:59.347882 22182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:59.360018 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:59.359956 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:59.360034 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:59.360503 22182 server_base.cc:1034] running on GCE node
I20260925 15:02:59.361424 22182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:59.362845 22182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:59.364068 22182 hybrid_clock.cc:648] HybridClock initialized: now 1790348579364009 us; error 52 us; skew 500 ppm
I20260925 15:02:59.364389 22182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:59 dist-test-slave-qxh0 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:59.367367 22182 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:02:59.368643 22182 webserver.cc:460] Webserver started at http://127.4.243.254:37749/ using document root <none> and password file <none>
I20260925 15:02:59.369262 22182 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:59.369355 22182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:59.369603 22182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:59.372366 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "6ca7e72937a047c880dac40cd90866ca"
format_stamp: "Formatted at 2026-09-25 15:02:59 on dist-test-slave-qxh0"
server_key: "15c53e55a40e80cbbec8456e0cdaf5ca"
server_key_iv: "3809b1cb4d07c7d74ebc8096e0a2533b"
server_key_version: "encryptionkey@0"
I20260925 15:02:59.373010 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "6ca7e72937a047c880dac40cd90866ca"
format_stamp: "Formatted at 2026-09-25 15:02:59 on dist-test-slave-qxh0"
server_key: "15c53e55a40e80cbbec8456e0cdaf5ca"
server_key_iv: "3809b1cb4d07c7d74ebc8096e0a2533b"
server_key_version: "encryptionkey@0"
I20260925 15:02:59.377641 22182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:02:59.380770 22197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:59.381942 22182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:02:59.382072 22182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "6ca7e72937a047c880dac40cd90866ca"
format_stamp: "Formatted at 2026-09-25 15:02:59 on dist-test-slave-qxh0"
server_key: "15c53e55a40e80cbbec8456e0cdaf5ca"
server_key_iv: "3809b1cb4d07c7d74ebc8096e0a2533b"
server_key_version: "encryptionkey@0"
I20260925 15:02:59.382203 22182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:59.406976 22182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:59.410753 22182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:59.411010 22182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:59.429208 22182 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:38917
I20260925 15:02:59.429239 22248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:38917 every 8 connection(s)
I20260925 15:02:59.430490 22182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:59.431988 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 22182
I20260925 15:02:59.432192 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:02:59.432857 5071 external_mini_cluster.cc:1428] Setting key 3fef147f8e24aae194e26f4426f0dfe0
I20260925 15:02:59.434757 22249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:02:59 dist-test-slave-qxh0 krb5kdc[22169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:02:59.448067 22249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca: Bootstrap starting.
I20260925 15:02:59.452054 22249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:59.453346 22249 log.cc:824] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:59.456617 22249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca: No bootstrap required, opened a new log
I20260925 15:02:59.457573 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:59.434928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.4.243.254:38917)
0925 15:02:59.435727 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:59.435741 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:59.435790 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:02:59.437861 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:59.437893 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:59.438412 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:59.438891 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:59.440272 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:59.442329 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:59.443693 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:59.444719 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:59.444917 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:59.449030 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:59.449041 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:59.449059 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:59.449106 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:59.453011 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:59.453753 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:59.453763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:59.453773 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:59.454018 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:59.454422 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:59.454425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:59.454427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:59.455114 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:59.455419 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:59.455801 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 15:02:59.456232 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":307,"threads_started":1}
I20260925 15:02:59.464805 22249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca7e72937a047c880dac40cd90866ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38917 } }
I20260925 15:02:59.465116 22249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:59.465176 22249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca7e72937a047c880dac40cd90866ca, State: Initialized, Role: FOLLOWER
I20260925 15:02:59.465699 22249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca7e72937a047c880dac40cd90866ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38917 } }
I20260925 15:02:59.465898 22249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:59.465991 22249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:59.466106 22249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:59.468119 22249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca7e72937a047c880dac40cd90866ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38917 } }
I20260925 15:02:59.468523 22249 leader_election.cc:302] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ca7e72937a047c880dac40cd90866ca; no voters:
I20260925 15:02:59.469169 22249 leader_election.cc:288] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:59.469298 22254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:59.470064 22254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [term 1 LEADER]: Becoming Leader. State: Replica: 6ca7e72937a047c880dac40cd90866ca, State: Running, Role: LEADER
I20260925 15:02:59.470731 22254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca7e72937a047c880dac40cd90866ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38917 } }
I20260925 15:02:59.471257 22249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:59.473611 22256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ca7e72937a047c880dac40cd90866ca. Latest consensus state: current_term: 1 leader_uuid: "6ca7e72937a047c880dac40cd90866ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca7e72937a047c880dac40cd90866ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38917 } } }
I20260925 15:02:59.473585 22255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ca7e72937a047c880dac40cd90866ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca7e72937a047c880dac40cd90866ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38917 } } }
I20260925 15:02:59.474103 22255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:59.474079 22256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:59.476253 22262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:59.480690 22262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:59.488976 22262 catalog_manager.cc:1347] Generated new cluster ID: e0755fb11c7a4c959d7c0b9473de7a45
I20260925 15:02:59.489073 22262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:59.504269 22262 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:59.505833 22262 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:59.514778 22262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca: Generated new TSK 0
I20260925 15:02:59.515503 22262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 15:02:59.582393 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38917
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:59.908910 22275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:59.909155 22275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:59.909303 22275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:59.916504 22275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:59.916601 22275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:59.916790 22275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:02:59.931525 22275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:02:59.933387 22275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:59.935191 22275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:59.949527 22283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:59.949473 22281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:59.949785 22280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:59.950793 22275 server_base.cc:1034] running on GCE node
I20260925 15:02:59.951557 22275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:59.952790 22275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:59.954083 22275 hybrid_clock.cc:648] HybridClock initialized: now 1790348579954032 us; error 64 us; skew 500 ppm
I20260925 15:02:59.954499 22275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:02:59 dist-test-slave-qxh0 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:59.959049 22275 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:02:59.960696 22275 webserver.cc:460] Webserver started at http://127.4.243.193:40035/ using document root <none> and password file <none>
I20260925 15:02:59.961419 22275 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:59.961555 22275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:59.961891 22275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:59.964906 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "a67d6447a1dc4bc5a27960d9662836e8"
format_stamp: "Formatted at 2026-09-25 15:02:59 on dist-test-slave-qxh0"
server_key: "54a5c4e97f64bd3ebf1e41eb564ea774"
server_key_iv: "5bfa3127c4fe1bc97b19e35bead6f1d2"
server_key_version: "encryptionkey@0"
I20260925 15:02:59.966032 22275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "a67d6447a1dc4bc5a27960d9662836e8"
format_stamp: "Formatted at 2026-09-25 15:02:59 on dist-test-slave-qxh0"
server_key: "54a5c4e97f64bd3ebf1e41eb564ea774"
server_key_iv: "5bfa3127c4fe1bc97b19e35bead6f1d2"
server_key_version: "encryptionkey@0"
I20260925 15:02:59.973668 22275 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260925 15:02:59.979437 22290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:59.981570 22275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:02:59.981745 22275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "a67d6447a1dc4bc5a27960d9662836e8"
format_stamp: "Formatted at 2026-09-25 15:02:59 on dist-test-slave-qxh0"
server_key: "54a5c4e97f64bd3ebf1e41eb564ea774"
server_key_iv: "5bfa3127c4fe1bc97b19e35bead6f1d2"
server_key_version: "encryptionkey@0"
I20260925 15:02:59.982004 22275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:00.015594 22275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:00.020494 22275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:00.020744 22275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:00.021581 22275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:00.022881 22275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:00.022950 22275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:00.023043 22275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:00.023085 22275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:00.055166 22275 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:35557
I20260925 15:03:00.055222 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:35557 every 8 connection(s)
I20260925 15:03:00.056521 22275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:03:00.059106 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 22275
I20260925 15:03:00.059324 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:03:00.060140 5071 external_mini_cluster.cc:1428] Setting key 7e8feec3554e971495346bc17c648d5e
Sep 25 15:03:00 dist-test-slave-qxh0 krb5kdc[22169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:00.076802 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:00.059244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:54283 (local address 127.4.243.254:38917)
0925 15:03:00.059771 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:00.059776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:00.061071 (+ 1295us) server_negotiation.cc:407] Connection header received
0925 15:03:00.062909 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:00.062915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:00.063004 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:00.063121 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:00.064556 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.065235 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.066835 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.067225 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.067356 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:00.070755 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:00.070773 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:00.070776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:00.070805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:00.073599 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:00.074296 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:00.074302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:00.074313 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:00.074452 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:00.074920 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:00.074925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:00.074929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:00.075234 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:00.075390 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:00.076322 (+ 932us) server_negotiation.cc:299] Negotiation successful
0925 15:03:00.076530 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":299,"threads_started":1}
I20260925 15:03:00.078245 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:00.059867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38917 (local address 127.4.243.193:54283)
0925 15:03:00.060669 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:00.060723 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:00.062650 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:00.063352 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:00.063360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:00.063843 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:00.064392 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:00.064408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.065392 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.065402 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:00.066693 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:00.066705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.067441 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.067449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:00.067671 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:00.068417 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:00.068437 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:00.070502 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:00.073734 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:00.073742 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:00.073757 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:00.074171 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:00.074581 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:00.074585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:00.074589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:00.074799 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:00.075404 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:00.075414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:00.075876 (+ 462us) client_negotiation.cc:769] Sending connection context
0925 15:03:00.076170 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 15:03:00.076774 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":362,"threads_started":1}
I20260925 15:03:00.079885 22403 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38917
I20260925 15:03:00.080202 22403 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:00.081008 22403 heartbeater.cc:511] Master 127.4.243.254:38917 requested a full tablet report, sending...
I20260925 15:03:00.083590 22214 ts_manager.cc:194] Registered new tserver with Master: a67d6447a1dc4bc5a27960d9662836e8 (127.4.243.193:35557)
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
I20260925 15:03:00.085249 22214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:54283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 15:03:00.113386 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38917
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:00.431064 22409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:00.431349 22409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:00.431507 22409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:00.438649 22409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:00.438815 22409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:00.439000 22409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:03:00.448896 22409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:00.450196 22409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:00.451520 22409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:00.464491 22415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:00.464638 22414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:00.464638 22417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:00.465224 22409 server_base.cc:1034] running on GCE node
I20260925 15:03:00.465828 22409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:00.466825 22409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:00.468052 22409 hybrid_clock.cc:648] HybridClock initialized: now 1790348580468013 us; error 54 us; skew 500 ppm
I20260925 15:03:00.468359 22409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:00 dist-test-slave-qxh0 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:00.471624 22409 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:03:00.473146 22409 webserver.cc:460] Webserver started at http://127.4.243.194:33355/ using document root <none> and password file <none>
I20260925 15:03:00.473773 22409 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:00.473915 22409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:00.474218 22409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:00.477174 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "5f277525122548cc88419e12ec81d4bf"
format_stamp: "Formatted at 2026-09-25 15:03:00 on dist-test-slave-qxh0"
server_key: "4c594404f4331ee927c1d4bc61fe7033"
server_key_iv: "22c4bb43b7534cb5bd24929a17ff5b62"
server_key_version: "encryptionkey@0"
I20260925 15:03:00.477916 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "5f277525122548cc88419e12ec81d4bf"
format_stamp: "Formatted at 2026-09-25 15:03:00 on dist-test-slave-qxh0"
server_key: "4c594404f4331ee927c1d4bc61fe7033"
server_key_iv: "22c4bb43b7534cb5bd24929a17ff5b62"
server_key_version: "encryptionkey@0"
I20260925 15:03:00.483148 22409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 15:03:00.486578 22424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:00.487854 22409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:03:00.488025 22409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "5f277525122548cc88419e12ec81d4bf"
format_stamp: "Formatted at 2026-09-25 15:03:00 on dist-test-slave-qxh0"
server_key: "4c594404f4331ee927c1d4bc61fe7033"
server_key_iv: "22c4bb43b7534cb5bd24929a17ff5b62"
server_key_version: "encryptionkey@0"
I20260925 15:03:00.488181 22409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:00.516547 22409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:00.520525 22409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:00.520807 22409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:00.521697 22409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:00.523082 22409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:00.523154 22409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:00.523243 22409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:00.523288 22409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:00.556102 22409 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:35203
I20260925 15:03:00.556190 22536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:35203 every 8 connection(s)
I20260925 15:03:00.557389 22409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:03:00.565619 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 22409
I20260925 15:03:00.565834 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:03:00.566294 5071 external_mini_cluster.cc:1428] Setting key 66736e2ede1934c30debfe964bd45a19
Sep 25 15:03:00 dist-test-slave-qxh0 krb5kdc[22169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:00.578529 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:00.560100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:48555 (local address 127.4.243.254:38917)
0925 15:03:00.560293 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:00.560297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:00.561332 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 15:03:00.563202 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:00.563208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:00.563301 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:00.563422 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:00.564829 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.565530 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.567377 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.567864 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.567998 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:00.572002 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:00.572011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:00.572018 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:00.572060 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:00.575500 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:00.576207 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:00.576215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:00.576221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:00.576381 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:00.576800 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:00.576807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:00.576812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:00.577170 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:00.577333 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:00.578139 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 15:03:00.578301 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:03:00.579994 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:00.560489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38917 (local address 127.4.243.194:48555)
0925 15:03:00.561170 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:00.561208 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:00.562933 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:00.563643 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:00.563652 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:00.564130 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:00.564663 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:00.564679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.565708 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.565716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:00.567226 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:00.567238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:00.568081 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:00.568088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:00.568401 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:00.569091 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:00.569123 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:00.571800 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:00.575658 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:00.575670 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:00.575681 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:00.576088 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:00.576481 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:00.576485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:00.576487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:00.576657 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:00.577312 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:00.577322 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:00.577770 (+ 448us) client_negotiation.cc:769] Sending connection context
0925 15:03:00.578069 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 15:03:00.578630 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":272,"threads_started":1}
I20260925 15:03:00.581359 22537 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38917
I20260925 15:03:00.581629 22537 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:00.582268 22537 heartbeater.cc:511] Master 127.4.243.254:38917 requested a full tablet report, sending...
I20260925 15:03:00.583657 22214 ts_manager.cc:194] Registered new tserver with Master: 5f277525122548cc88419e12ec81d4bf (127.4.243.194:35203)
I20260925 15:03:00.584465 22214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:48555
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 15:03:00.621977 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38917
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:00.941648 22542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:00.941955 22542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:00.942121 22542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:00.949321 22542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:00.949465 22542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:00.949633 22542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:03:00.958994 22542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:00.960325 22542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:00.961613 22542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:00.974572 22550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:00.974536 22548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:00.974633 22547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:00.975347 22542 server_base.cc:1034] running on GCE node
I20260925 15:03:00.976027 22542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:00.976924 22542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:00.978124 22542 hybrid_clock.cc:648] HybridClock initialized: now 1790348580978084 us; error 26 us; skew 500 ppm
I20260925 15:03:00.978456 22542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:00 dist-test-slave-qxh0 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:00.981561 22542 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:03:00.982935 22542 webserver.cc:460] Webserver started at http://127.4.243.195:32945/ using document root <none> and password file <none>
I20260925 15:03:00.983551 22542 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:00.983649 22542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:00.983897 22542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:00.986804 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "89fe667dd8f14961ac29db759c8396b0"
format_stamp: "Formatted at 2026-09-25 15:03:00 on dist-test-slave-qxh0"
server_key: "8e6cb5f4730f81409ceea79d9e7ca05d"
server_key_iv: "54debe5aca74c3ad6f9da08fc45d0a31"
server_key_version: "encryptionkey@0"
I20260925 15:03:00.987515 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "89fe667dd8f14961ac29db759c8396b0"
format_stamp: "Formatted at 2026-09-25 15:03:00 on dist-test-slave-qxh0"
server_key: "8e6cb5f4730f81409ceea79d9e7ca05d"
server_key_iv: "54debe5aca74c3ad6f9da08fc45d0a31"
server_key_version: "encryptionkey@0"
I20260925 15:03:00.992635 22542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:00.995990 22557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:00.997347 22542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:03:00.997536 22542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "89fe667dd8f14961ac29db759c8396b0"
format_stamp: "Formatted at 2026-09-25 15:03:00 on dist-test-slave-qxh0"
server_key: "8e6cb5f4730f81409ceea79d9e7ca05d"
server_key_iv: "54debe5aca74c3ad6f9da08fc45d0a31"
server_key_version: "encryptionkey@0"
I20260925 15:03:00.997753 22542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:01.020329 22542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:01.024654 22542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:01.024972 22542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:01.025959 22542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:01.027294 22542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:01.027372 22542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:01.027499 22542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:01.027549 22542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:01.062853 22542 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:35771
I20260925 15:03:01.062922 22669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:35771 every 8 connection(s)
I20260925 15:03:01.064020 22542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:03:01.072440 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 22542
I20260925 15:03:01.072623 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:03:01.073055 5071 external_mini_cluster.cc:1428] Setting key a4469fde5925ab6ab6c48db7b4568a77
Sep 25 15:03:01 dist-test-slave-qxh0 krb5kdc[22169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:01.083295 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.066380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:35015 (local address 127.4.243.254:38917)
0925 15:03:01.066623 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.066629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.067711 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 15:03:01.069442 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.069447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.069525 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.069615 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:01.071131 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.071954 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.073705 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.074294 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.074419 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.078117 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:01.078122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:01.078125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:01.078159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:01.080639 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.081359 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.081363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.081366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.081490 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.081851 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.081854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.081856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.082092 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:01.082204 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.082880 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.083042 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 15:03:01.084228 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.066875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38917 (local address 127.4.243.195:35015)
0925 15:03:01.067548 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:01.067581 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:01.069188 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:01.069802 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:01.069809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:01.070287 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:01.070976 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.070999 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.072078 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.072082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.073546 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.073561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.074483 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.074488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.074888 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.075591 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:01.075619 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:01.077923 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:01.080820 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:01.080826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:01.080839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:01.081247 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:01.081582 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:01.081585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:01.081587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:01.081755 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:01.082204 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:01.082210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:01.082501 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:03:01.082791 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 15:03:01.083156 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260925 15:03:01.085600 22670 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38917
I20260925 15:03:01.085930 22670 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:01.086887 22670 heartbeater.cc:511] Master 127.4.243.254:38917 requested a full tablet report, sending...
I20260925 15:03:01.088493 22403 heartbeater.cc:503] Master 127.4.243.254:38917 was elected leader, sending a full tablet report...
I20260925 15:03:01.088524 22213 ts_manager.cc:194] Registered new tserver with Master: 89fe667dd8f14961ac29db759c8396b0 (127.4.243.195:35771)
I20260925 15:03:01.089617 22213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:35015
I20260925 15:03:01.100423 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:03:01.117177 22405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.104801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.4.243.254:38917)
0925 15:03:01.104949 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.104954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.105412 (+ 458us) server_negotiation.cc:407] Connection header received
0925 15:03:01.105569 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.105573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.105631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.105768 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:01.107293 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.107998 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.109389 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.109783 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.109875 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.112182 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:01.112186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:01.112189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:01.112218 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:01.114539 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.115193 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.115197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.115199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.115362 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.115759 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.115766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.115771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.116066 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:01.116328 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.116655 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.116872 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:03:01.121281 22213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:03:01.128036 22213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:03:01.149717 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.139533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:35557 (local address 127.0.0.1:41066)
0925 15:03:01.140329 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:01.140382 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:01.140577 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:01.141952 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:01.141962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:01.141997 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:01.142424 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.142440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.144627 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.144633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.147035 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.147054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.148627 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.148633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.148835 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.148884 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:03:01.149018 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 15:03:01.149229 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":35,"thread_start_us":297,"threads_started":1}
I20260925 15:03:01.150152 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.140180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:35771 (local address 127.0.0.1:43734)
0925 15:03:01.141614 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:01.141636 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:01.141740 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:01.143367 (+ 1627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:01.143372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:01.143396 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:01.143738 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.143749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.145483 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.145487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.147602 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.147621 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.149315 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.149320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.149487 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.149530 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:03:01.149591 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:03:01.149752 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"mutex_wait_us":6,"thread_start_us":1200,"threads_started":1}
I20260925 15:03:01.150563 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.139764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.4.243.193:35557)
0925 15:03:01.141273 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.141284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.141317 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:03:01.141425 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.141434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.141715 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.142657 (+ 942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:01.142905 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.144499 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.147213 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.148439 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.148574 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.148819 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.149073 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.150239 (+ 1166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1340,"thread_start_us":1170,"threads_started":1}
I20260925 15:03:01.151264 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.141101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.4.243.194:35203)
0925 15:03:01.142260 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.142269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.143355 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 15:03:01.143457 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.143466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.143706 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.143914 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:01.144439 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.146029 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.148283 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.149643 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.149790 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.149963 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.150635 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.150956 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":816,"threads_started":1}
I20260925 15:03:01.151299 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.140805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.4.243.195:35771)
0925 15:03:01.142759 (+ 1954us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.142766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.142798 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:03:01.142888 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.142894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.143135 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.143908 (+ 773us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:01.144158 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.145338 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.147768 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.149078 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.150507 (+ 1429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.150765 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.150841 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.150974 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1771,"thread_start_us":1617,"threads_started":1}
I20260925 15:03:01.151597 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.140595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:35203 (local address 127.0.0.1:58290)
0925 15:03:01.143056 (+ 2461us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:01.143089 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:01.143208 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:01.143895 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:01.143900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:01.143925 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:01.144282 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.144290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.146195 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.146201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.148110 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.148137 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.150118 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.150125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.150298 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.150336 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:03:01.150423 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:03:01.150537 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2336,"thread_start_us":2215,"threads_started":1}
I20260925 15:03:01.153961 22338 tablet_service.cc:1467] Processing CreateTablet for tablet 079fedf5f0ea4cdeae7af09a6e04043f (DEFAULT_TABLE table=test-table [id=643a4c7a73194591918590a5925dd69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:01.154636 22605 tablet_service.cc:1467] Processing CreateTablet for tablet 079fedf5f0ea4cdeae7af09a6e04043f (DEFAULT_TABLE table=test-table [id=643a4c7a73194591918590a5925dd69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:01.154636 22472 tablet_service.cc:1467] Processing CreateTablet for tablet 079fedf5f0ea4cdeae7af09a6e04043f (DEFAULT_TABLE table=test-table [id=643a4c7a73194591918590a5925dd69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:01.155331 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079fedf5f0ea4cdeae7af09a6e04043f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:01.155929 22605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079fedf5f0ea4cdeae7af09a6e04043f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:01.155931 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 079fedf5f0ea4cdeae7af09a6e04043f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:01.166780 22688 tablet_bootstrap.cc:492] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Bootstrap starting.
I20260925 15:03:01.170310 22690 tablet_bootstrap.cc:492] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Bootstrap starting.
I20260925 15:03:01.170583 22688 tablet_bootstrap.cc:654] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:01.171403 22689 tablet_bootstrap.cc:492] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Bootstrap starting.
I20260925 15:03:01.172365 22688 log.cc:824] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:01.174378 22690 tablet_bootstrap.cc:654] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:01.174691 22689 tablet_bootstrap.cc:654] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:01.175570 22688 tablet_bootstrap.cc:492] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: No bootstrap required, opened a new log
I20260925 15:03:01.175654 22690 log.cc:824] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:01.175676 22689 log.cc:824] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:01.175827 22688 ts_tablet_manager.cc:1397] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:03:01.178399 22690 tablet_bootstrap.cc:492] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: No bootstrap required, opened a new log
I20260925 15:03:01.178606 22689 tablet_bootstrap.cc:492] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: No bootstrap required, opened a new log
I20260925 15:03:01.178738 22690 ts_tablet_manager.cc:1397] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:03:01.178876 22689 ts_tablet_manager.cc:1397] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 15:03:01.186266 22690 raft_consensus.cc:348] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.186333 22688 raft_consensus.cc:348] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.186733 22690 raft_consensus.cc:374] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:01.186736 22688 raft_consensus.cc:374] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:01.186833 22688 raft_consensus.cc:729] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67d6447a1dc4bc5a27960d9662836e8, State: Initialized, Role: FOLLOWER
I20260925 15:03:01.186828 22690 raft_consensus.cc:729] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fe667dd8f14961ac29db759c8396b0, State: Initialized, Role: FOLLOWER
I20260925 15:03:01.187454 22688 consensus_queue.cc:260] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.187495 22690 consensus_queue.cc:260] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.189239 22670 heartbeater.cc:503] Master 127.4.243.254:38917 was elected leader, sending a full tablet report...
I20260925 15:03:01.189222 22689 raft_consensus.cc:348] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.189567 22690 ts_tablet_manager.cc:1428] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 15:03:01.189605 22689 raft_consensus.cc:374] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:01.189714 22689 raft_consensus.cc:729] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f277525122548cc88419e12ec81d4bf, State: Initialized, Role: FOLLOWER
I20260925 15:03:01.190219 22689 consensus_queue.cc:260] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.191359 22537 heartbeater.cc:503] Master 127.4.243.254:38917 was elected leader, sending a full tablet report...
I20260925 15:03:01.191921 22689 ts_tablet_manager.cc:1428] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 15:03:01.192586 22688 ts_tablet_manager.cc:1428] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260925 15:03:01.213980 22694 raft_consensus.cc:482] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:03:01.214254 22694 raft_consensus.cc:504] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.215876 22694 leader_election.cc:288] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f277525122548cc88419e12ec81d4bf (127.4.243.194:35203), 89fe667dd8f14961ac29db759c8396b0 (127.4.243.195:35771)
I20260925 15:03:01.223042 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.216422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:37755 (local address 127.4.243.194:35203)
0925 15:03:01.216630 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.216638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.217243 (+ 605us) server_negotiation.cc:407] Connection header received
0925 15:03:01.217457 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.217461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.217519 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.217615 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:01.218354 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.219142 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.221316 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.222049 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.222224 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.222291 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.222677 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.222801 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:03:01.223040 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.216375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:34571 (local address 127.4.243.195:35771)
0925 15:03:01.216579 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.216587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.217338 (+ 751us) server_negotiation.cc:407] Connection header received
0925 15:03:01.217528 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.217532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.217585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.217673 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:01.218323 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.219153 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.221285 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.222027 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.222172 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.222255 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.222655 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.222786 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:03:01.223191 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.216341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:35203 (local address 127.4.243.193:37755)
0925 15:03:01.217127 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:01.217142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:01.217315 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:01.217636 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:01.217640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:01.217663 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:01.218213 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.218223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.219280 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.219287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.221143 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.221155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.222304 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.222312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.222463 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.222515 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:03:01.222650 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:03:01.222772 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":58,"spinlock_wait_cycles":22272,"thread_start_us":455,"threads_started":1}
I20260925 15:03:01.223533 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.216435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:35771 (local address 127.4.243.193:34571)
0925 15:03:01.217227 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:01.217240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:01.217390 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:01.217696 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:01.217699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:01.217724 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:01.218157 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.218165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.219309 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.219313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.221129 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:01.221140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.222242 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.222248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:01.222414 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.222504 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 15:03:01.222576 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:03:01.222747 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":111,"thread_start_us":405,"threads_started":1}
I20260925 15:03:01.224233 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079fedf5f0ea4cdeae7af09a6e04043f" candidate_uuid: "a67d6447a1dc4bc5a27960d9662836e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f277525122548cc88419e12ec81d4bf" is_pre_election: true
I20260925 15:03:01.224453 22625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079fedf5f0ea4cdeae7af09a6e04043f" candidate_uuid: "a67d6447a1dc4bc5a27960d9662836e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89fe667dd8f14961ac29db759c8396b0" is_pre_election: true
I20260925 15:03:01.224658 22492 raft_consensus.cc:2454] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a67d6447a1dc4bc5a27960d9662836e8 in term 0.
I20260925 15:03:01.224967 22625 raft_consensus.cc:2454] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a67d6447a1dc4bc5a27960d9662836e8 in term 0.
I20260925 15:03:01.225433 22294 leader_election.cc:302] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f277525122548cc88419e12ec81d4bf, a67d6447a1dc4bc5a27960d9662836e8; no voters:
I20260925 15:03:01.225852 22694 raft_consensus.cc:2781] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:03:01.226003 22694 raft_consensus.cc:482] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:03:01.226119 22694 raft_consensus.cc:3037] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:01.228268 22694 raft_consensus.cc:504] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.228833 22694 leader_election.cc:288] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [CANDIDATE]: Term 1 election: Requested vote from peers 5f277525122548cc88419e12ec81d4bf (127.4.243.194:35203), 89fe667dd8f14961ac29db759c8396b0 (127.4.243.195:35771)
I20260925 15:03:01.229240 22492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079fedf5f0ea4cdeae7af09a6e04043f" candidate_uuid: "a67d6447a1dc4bc5a27960d9662836e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f277525122548cc88419e12ec81d4bf"
I20260925 15:03:01.229421 22492 raft_consensus.cc:3037] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:01.229351 22625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "079fedf5f0ea4cdeae7af09a6e04043f" candidate_uuid: "a67d6447a1dc4bc5a27960d9662836e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89fe667dd8f14961ac29db759c8396b0"
I20260925 15:03:01.229604 22625 raft_consensus.cc:3037] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:01.231657 22492 raft_consensus.cc:2454] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a67d6447a1dc4bc5a27960d9662836e8 in term 1.
I20260925 15:03:01.232048 22625 raft_consensus.cc:2454] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a67d6447a1dc4bc5a27960d9662836e8 in term 1.
I20260925 15:03:01.232040 22294 leader_election.cc:302] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f277525122548cc88419e12ec81d4bf, a67d6447a1dc4bc5a27960d9662836e8; no voters:
I20260925 15:03:01.232340 22694 raft_consensus.cc:2781] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:01.232784 22694 raft_consensus.cc:686] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 1 LEADER]: Becoming Leader. State: Replica: a67d6447a1dc4bc5a27960d9662836e8, State: Running, Role: LEADER
I20260925 15:03:01.233209 22694 consensus_queue.cc:237] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } }
I20260925 15:03:01.238095 22214 catalog_manager.cc:5495] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a67d6447a1dc4bc5a27960d9662836e8 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "a67d6447a1dc4bc5a27960d9662836e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a67d6447a1dc4bc5a27960d9662836e8" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20260925 15:03:01.275794 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.270364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.4.243.193:35557)
0925 15:03:01.270578 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.270584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.270709 (+ 125us) server_negotiation.cc:407] Connection header received
0925 15:03:01.271079 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.271083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.271162 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.271310 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:01.271627 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.272374 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.273736 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.274130 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.274244 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.274686 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:01.275260 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:01.275373 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.275437 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.275500 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:03:01.284286 22625 raft_consensus.cc:1261] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 1 FOLLOWER]: Refusing update from remote peer a67d6447a1dc4bc5a27960d9662836e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:01.284283 22492 raft_consensus.cc:1261] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 1 FOLLOWER]: Refusing update from remote peer a67d6447a1dc4bc5a27960d9662836e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:01.285291 22699 consensus_queue.cc:1035] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f277525122548cc88419e12ec81d4bf" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:01.285657 22694 consensus_queue.cc:1035] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89fe667dd8f14961ac29db759c8396b0" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:01.297554 22704 mvcc.cc:205] Tried to move back new op lower bound from 7333267788930363392 to 7333267788734603264. Current Snapshot: MvccSnapshot[applied={T|T < 7333267788930363392}]
I20260925 15:03:01.306774 22703 mvcc.cc:205] Tried to move back new op lower bound from 7333267788930363392 to 7333267788734603264. Current Snapshot: MvccSnapshot[applied={T|T < 7333267788930363392 or (T in {7333267788930363392})}]
I20260925 15:03:01.311941 22212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:01.312247 22212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:01.316155 22212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ca7e72937a047c880dac40cd90866ca: Sending DeleteTablet for 3 replicas of tablet 079fedf5f0ea4cdeae7af09a6e04043f
I20260925 15:03:01.317231 22338 tablet_service.cc:1514] Processing DeleteTablet for tablet 079fedf5f0ea4cdeae7af09a6e04043f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:01 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:41066
I20260925 15:03:01.317425 22472 tablet_service.cc:1514] Processing DeleteTablet for tablet 079fedf5f0ea4cdeae7af09a6e04043f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:01 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:58290
I20260925 15:03:01.317561 22605 tablet_service.cc:1514] Processing DeleteTablet for tablet 079fedf5f0ea4cdeae7af09a6e04043f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:01 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:43734
I20260925 15:03:01.320269 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 22275
I20260925 15:03:01.326912 22712 tablet_replica.cc:330] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: stopping tablet replica
I20260925 15:03:01.327471 22712 raft_consensus.cc:2229] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:01.328119 22712 raft_consensus.cc:2258] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:01.331988 22713 tablet_replica.cc:330] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: stopping tablet replica
I20260925 15:03:01.332500 22713 raft_consensus.cc:2229] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:01.333069 22713 raft_consensus.cc:2258] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:01.335418 22713 ts_tablet_manager.cc:1905] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:01.341467 22712 ts_tablet_manager.cc:1905] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:01.342247 22713 ts_tablet_manager.cc:1918] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:01.342399 22713 log.cc:1196] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/079fedf5f0ea4cdeae7af09a6e04043f
I20260925 15:03:01.342955 22713 ts_tablet_manager.cc:1939] T 079fedf5f0ea4cdeae7af09a6e04043f P 89fe667dd8f14961ac29db759c8396b0: Deleting consensus metadata
I20260925 15:03:01.344724 22199 catalog_manager.cc:4841] TS 89fe667dd8f14961ac29db759c8396b0 (127.4.243.195:35771): tablet 079fedf5f0ea4cdeae7af09a6e04043f (table test-table [id=643a4c7a73194591918590a5925dd69b]) successfully deleted
I20260925 15:03:01.345638 22711 tablet_replica.cc:330] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: stopping tablet replica
I20260925 15:03:01.346112 22711 raft_consensus.cc:2229] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:03:01.346832 22711 raft_consensus.cc:2258] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 15:03:01 dist-test-slave-qxh0 krb5kdc[22169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 15:03:01.348791 22711 ts_tablet_manager.cc:1905] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:01.352517 22712 ts_tablet_manager.cc:1918] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:01.352721 22712 log.cc:1196] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/079fedf5f0ea4cdeae7af09a6e04043f
I20260925 15:03:01.353283 22712 ts_tablet_manager.cc:1939] T 079fedf5f0ea4cdeae7af09a6e04043f P 5f277525122548cc88419e12ec81d4bf: Deleting consensus metadata
I20260925 15:03:01.355087 22201 catalog_manager.cc:4841] TS 5f277525122548cc88419e12ec81d4bf (127.4.243.194:35203): tablet 079fedf5f0ea4cdeae7af09a6e04043f (table test-table [id=643a4c7a73194591918590a5925dd69b]) successfully deleted
I20260925 15:03:01.356397 22711 ts_tablet_manager.cc:1918] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:01.356602 22711 log.cc:1196] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/079fedf5f0ea4cdeae7af09a6e04043f
I20260925 15:03:01.357159 22711 ts_tablet_manager.cc:1939] T 079fedf5f0ea4cdeae7af09a6e04043f P a67d6447a1dc4bc5a27960d9662836e8: Deleting consensus metadata
I20260925 15:03:01.358659 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.321970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.4.243.193:35557)
0925 15:03:01.322226 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.322235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.334164 (+ 11929us) server_negotiation.cc:407] Connection header received
0925 15:03:01.334659 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.334673 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.334760 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.334837 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:01.336677 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.337889 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.341676 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.342443 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.342555 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.349887 (+ 7332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:01.349903 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:01.349924 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:01.349972 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:01.354213 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.355714 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.355723 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.355727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.355907 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.356376 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.356380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.356387 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.356824 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:01.357124 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.357834 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.358110 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:03:01.358963 22201 catalog_manager.cc:4841] TS a67d6447a1dc4bc5a27960d9662836e8 (127.4.243.193:35557): tablet 079fedf5f0ea4cdeae7af09a6e04043f (table test-table [id=643a4c7a73194591918590a5925dd69b]) successfully deleted
I20260925 15:03:01.359344 22398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:01.460897 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 22275
I20260925 15:03:01.479846 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 22409
Sep 25 15:03:01 dist-test-slave-qxh0 krb5kdc[22169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:03:01.494273 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.480553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.4.243.194:35203)
0925 15:03:01.480720 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.480724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.480741 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:03:01.480892 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.480896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.480967 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.481062 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:01.482293 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.483074 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.484360 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.484720 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.484838 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.489194 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:01.489207 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:01.489222 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:01.489250 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:01.492006 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.492659 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.492665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.492667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.492792 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.493151 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.493153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.493155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.493466 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:01.493677 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.493892 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.494036 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:03:01.494982 22532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:01.582206 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 22409
I20260925 15:03:01.599020 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 22542
Sep 25 15:03:01 dist-test-slave-qxh0 krb5kdc[22169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 15:03:01.613559 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:01.599788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.4.243.195:35771)
0925 15:03:01.599954 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:01.599958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:01.599978 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:03:01.600070 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:01.600074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:01.600137 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:01.600231 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:01.601482 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.602205 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.603557 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:01.603966 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:01.604097 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:01.608302 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:01.608317 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:01.608333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:01.608369 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:01.610936 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.611564 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.611572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.611574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.611706 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:01.612058 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:01.612064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:01.612067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:01.612405 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:01.612693 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:01.613028 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:03:01.613261 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:03:01.614168 22665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:01.702255 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 22542
I20260925 15:03:01.719116 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 22182
I20260925 15:03:01.720611 22244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:01.831346 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 22182
2026-09-25T15:03:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:01Z Disabled control of system clock
I20260925 15:03:01.908171 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38211
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:03:02.227310 22722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:02.227572 22722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:02.227768 22722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:02.234648 22722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:02.234750 22722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:02.234851 22722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:02.234966 22722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:02.235023 22722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:02.244032 22722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:38211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:02.245633 22722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:02.247675 22722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:02.259964 22727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:02.260176 22728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:02.260006 22730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:02.261193 22722 server_base.cc:1034] running on GCE node
I20260925 15:03:02.261965 22722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:02.263499 22722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:02.264783 22722 hybrid_clock.cc:648] HybridClock initialized: now 1790348582264748 us; error 57 us; skew 500 ppm
I20260925 15:03:02.265156 22722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:02.267578 22722 webserver.cc:460] Webserver started at http://127.4.243.254:34231/ using document root <none> and password file <none>
I20260925 15:03:02.268297 22722 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:02.268438 22722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:02.268790 22722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:02.271850 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "98cf2adfc08b4407a6aa5e1b80063615"
format_stamp: "Formatted at 2026-09-25 15:03:02 on dist-test-slave-qxh0"
server_key: "a56e5fc1f532fb14c65d07513b826a5a"
server_key_iv: "5902d057c469c7a2fd8ec7f9d62cd2cf"
server_key_version: "encryptionkey@0"
I20260925 15:03:02.272677 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "98cf2adfc08b4407a6aa5e1b80063615"
format_stamp: "Formatted at 2026-09-25 15:03:02 on dist-test-slave-qxh0"
server_key: "a56e5fc1f532fb14c65d07513b826a5a"
server_key_iv: "5902d057c469c7a2fd8ec7f9d62cd2cf"
server_key_version: "encryptionkey@0"
I20260925 15:03:02.277978 22722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 15:03:02.282112 22736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:02.283507 22722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:02.283721 22722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "98cf2adfc08b4407a6aa5e1b80063615"
format_stamp: "Formatted at 2026-09-25 15:03:02 on dist-test-slave-qxh0"
server_key: "a56e5fc1f532fb14c65d07513b826a5a"
server_key_iv: "5902d057c469c7a2fd8ec7f9d62cd2cf"
server_key_version: "encryptionkey@0"
I20260925 15:03:02.283950 22722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:02.317655 22722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:02.318581 22722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:02.318944 22722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:02.338532 22722 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:38211
I20260925 15:03:02.338551 22787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:38211 every 8 connection(s)
I20260925 15:03:02.340006 22722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:02.343753 22788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:02.350134 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 22722
I20260925 15:03:02.350324 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:02.350910 5071 external_mini_cluster.cc:1428] Setting key 8f4475ebdf18d13eec772d7b11a84070
I20260925 15:03:02.354552 22788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615: Bootstrap starting.
I20260925 15:03:02.357743 22788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:02.358925 22788 log.cc:824] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:02.361696 22788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615: No bootstrap required, opened a new log
I20260925 15:03:02.368942 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:02.352956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.4.243.254:38211)
0925 15:03:02.353876 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:02.353887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:02.353928 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:03:02.355782 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:02.355806 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:02.356298 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:02.356692 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:02.357316 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:02.358888 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:02.360422 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:02.361154 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:02.361376 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:02.366401 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:02.366409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:02.366426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:02.367077 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:02.367160 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:02.367181 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:02.367371 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:02.367615 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:03:02.367950 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":323,"threads_started":1}
I20260925 15:03:02.369655 22788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cf2adfc08b4407a6aa5e1b80063615" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38211 } }
I20260925 15:03:02.370085 22788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:02.370260 22788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cf2adfc08b4407a6aa5e1b80063615, State: Initialized, Role: FOLLOWER
I20260925 15:03:02.370895 22788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cf2adfc08b4407a6aa5e1b80063615" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38211 } }
I20260925 15:03:02.371169 22788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:02.371322 22788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:02.371519 22788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:02.373730 22788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cf2adfc08b4407a6aa5e1b80063615" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38211 } }
I20260925 15:03:02.374248 22788 leader_election.cc:302] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98cf2adfc08b4407a6aa5e1b80063615; no voters:
I20260925 15:03:02.374899 22788 leader_election.cc:288] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:02.375036 22793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:02.376494 22793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [term 1 LEADER]: Becoming Leader. State: Replica: 98cf2adfc08b4407a6aa5e1b80063615, State: Running, Role: LEADER
I20260925 15:03:02.377012 22788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:02.376968 22793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cf2adfc08b4407a6aa5e1b80063615" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38211 } }
I20260925 15:03:02.379900 22795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98cf2adfc08b4407a6aa5e1b80063615. Latest consensus state: current_term: 1 leader_uuid: "98cf2adfc08b4407a6aa5e1b80063615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cf2adfc08b4407a6aa5e1b80063615" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38211 } } }
I20260925 15:03:02.380364 22795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:02.379899 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98cf2adfc08b4407a6aa5e1b80063615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cf2adfc08b4407a6aa5e1b80063615" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 38211 } } }
I20260925 15:03:02.381018 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615 [sys.catalog]: This master's current role is: LEADER
W20260925 15:03:02.389945 22809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:03:02.390045 22809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:03:02.390175 22800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:02.392416 22800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:02.401087 22800 catalog_manager.cc:1347] Generated new cluster ID: b18af1fce1c2485aa977b4d0fbdb9aa7
I20260925 15:03:02.401280 22800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:02.411370 22800 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:02.412892 22800 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:02.425487 22800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615: Generated new TSK 0
I20260925 15:03:02.426334 22800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:02.436049 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38211
--builtin_ntp_servers=127.4.243.212:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:03:02.780758 22812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:02.781008 22812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:02.781142 22812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:02.788798 22812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:02.788884 22812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:02.789093 22812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:03:02.797757 22812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:02.799078 22812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:02.800844 22812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:02.813733 22820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:02.813756 22818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:02.813977 22817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:02.814733 22812 server_base.cc:1034] running on GCE node
I20260925 15:03:02.815280 22812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:02.816176 22812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:02.817404 22812 hybrid_clock.cc:648] HybridClock initialized: now 1790348582817362 us; error 58 us; skew 500 ppm
I20260925 15:03:02.817727 22812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:02.819957 22812 webserver.cc:460] Webserver started at http://127.4.243.193:46323/ using document root <none> and password file <none>
I20260925 15:03:02.820616 22812 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:02.820757 22812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:02.821072 22812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:02.823930 22812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "8b3eeea8c28e4d96b8950883f3ebdad9"
format_stamp: "Formatted at 2026-09-25 15:03:02 on dist-test-slave-qxh0"
server_key: "819262b20c78e182a0841184902b6f56"
server_key_iv: "c890c207130b02e3e9e0c183bc55a9b3"
server_key_version: "encryptionkey@0"
I20260925 15:03:02.824674 22812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "8b3eeea8c28e4d96b8950883f3ebdad9"
format_stamp: "Formatted at 2026-09-25 15:03:02 on dist-test-slave-qxh0"
server_key: "819262b20c78e182a0841184902b6f56"
server_key_iv: "c890c207130b02e3e9e0c183bc55a9b3"
server_key_version: "encryptionkey@0"
I20260925 15:03:02.829541 22812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 15:03:02.832834 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:02.834101 22812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:03:02.834292 22812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "8b3eeea8c28e4d96b8950883f3ebdad9"
format_stamp: "Formatted at 2026-09-25 15:03:02 on dist-test-slave-qxh0"
server_key: "819262b20c78e182a0841184902b6f56"
server_key_iv: "c890c207130b02e3e9e0c183bc55a9b3"
server_key_version: "encryptionkey@0"
I20260925 15:03:02.834511 22812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:02.849732 22812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:02.850510 22812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:02.850836 22812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:02.851724 22812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:02.853158 22812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:02.853282 22812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:02.853410 22812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:02.853497 22812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:02.885412 22812 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:36963
I20260925 15:03:02.885464 22938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:36963 every 8 connection(s)
I20260925 15:03:02.886787 22812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:03:02.893508 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 22812
I20260925 15:03:02.893667 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:03:02.894210 5071 external_mini_cluster.cc:1428] Setting key abb848982652cba88aae3baeba01457c
I20260925 15:03:02.896960 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38211
--builtin_ntp_servers=127.4.243.212:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:03:02.900142 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:02.889405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:51851 (local address 127.4.243.254:38211)
0925 15:03:02.889860 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:02.889865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:02.890944 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 15:03:02.892687 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:02.892692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:02.892786 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:02.892901 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:02.894286 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:02.895267 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:02.897062 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:02.897747 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:02.897877 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:02.899195 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:02.899203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:02.899208 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:02.899233 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:02.899296 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:02.899314 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:02.899405 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:02.899610 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 15:03:02.899714 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260925 15:03:02.900954 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:02.889911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38211 (local address 127.4.243.193:51851)
0925 15:03:02.890799 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:02.890833 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:02.892426 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:02.893120 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:02.893128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:02.893268 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:02.894042 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:02.894066 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:02.895403 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:02.895409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:02.896904 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:02.896916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:02.897955 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:02.897960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:02.898142 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:02.898906 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:02.898908 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:02.898944 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:02.899013 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:02.899408 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:02.899416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:02.899421 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:03:02.899564 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:03:02.899849 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":262,"threads_started":1}
I20260925 15:03:02.902534 22939 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38211
I20260925 15:03:02.902828 22939 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:02.903524 22939 heartbeater.cc:511] Master 127.4.243.254:38211 requested a full tablet report, sending...
I20260925 15:03:02.905373 22753 ts_manager.cc:194] Registered new tserver with Master: 8b3eeea8c28e4d96b8950883f3ebdad9 (127.4.243.193:36963)
I20260925 15:03:02.906860 22753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.193:51851
W20260925 15:03:03.215806 22943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:03.216100 22943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:03.216255 22943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:03.223311 22943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:03.223426 22943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:03.223600 22943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:03:03.232276 22943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:03.233510 22943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:03.235261 22943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:03.248433 22948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:03.248353 22949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:03.248409 22951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:03.248962 22943 server_base.cc:1034] running on GCE node
I20260925 15:03:03.249667 22943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:03.250584 22943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:03.251854 22943 hybrid_clock.cc:648] HybridClock initialized: now 1790348583251818 us; error 60 us; skew 500 ppm
I20260925 15:03:03.252210 22943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:03.254310 22943 webserver.cc:460] Webserver started at http://127.4.243.194:41207/ using document root <none> and password file <none>
I20260925 15:03:03.255028 22943 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:03.255143 22943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:03.255450 22943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:03.258059 22943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "a45196745c0b45b2b2b2899ab58799af"
format_stamp: "Formatted at 2026-09-25 15:03:03 on dist-test-slave-qxh0"
server_key: "8ac46721d56a7c9c9f30cb26657340b8"
server_key_iv: "4dc041a3e5333e25ed160cfe687cce0c"
server_key_version: "encryptionkey@0"
I20260925 15:03:03.258869 22943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "a45196745c0b45b2b2b2899ab58799af"
format_stamp: "Formatted at 2026-09-25 15:03:03 on dist-test-slave-qxh0"
server_key: "8ac46721d56a7c9c9f30cb26657340b8"
server_key_iv: "4dc041a3e5333e25ed160cfe687cce0c"
server_key_version: "encryptionkey@0"
I20260925 15:03:03.263487 22943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:03:03.266705 22957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:03.267876 22943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:03:03.268040 22943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "a45196745c0b45b2b2b2899ab58799af"
format_stamp: "Formatted at 2026-09-25 15:03:03 on dist-test-slave-qxh0"
server_key: "8ac46721d56a7c9c9f30cb26657340b8"
server_key_iv: "4dc041a3e5333e25ed160cfe687cce0c"
server_key_version: "encryptionkey@0"
I20260925 15:03:03.268254 22943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:03.280794 22943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:03.281672 22943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:03.281962 22943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:03.282963 22943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:03.284401 22943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:03.284518 22943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:03.284646 22943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:03.284730 22943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:03.316452 22943 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:46561
I20260925 15:03:03.316505 23069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:46561 every 8 connection(s)
I20260925 15:03:03.317772 22943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:03:03.320379 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 22943
I20260925 15:03:03.320561 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:03:03.321003 5071 external_mini_cluster.cc:1428] Setting key a0ee4d0bff4056b6b51ae10c4f596a92
I20260925 15:03:03.324664 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38211
--builtin_ntp_servers=127.4.243.212:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:03:03.331471 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.320393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:52611 (local address 127.4.243.254:38211)
0925 15:03:03.320613 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:03.320620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:03.322227 (+ 1607us) server_negotiation.cc:407] Connection header received
0925 15:03:03.324493 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:03.324500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:03.324578 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:03.324702 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:03.325781 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.326963 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.328472 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.329195 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.329334 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.330621 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:03.330629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:03.330637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:03.330662 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:03.330715 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:03.330733 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:03.330824 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:03.331077 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:03:03.331176 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:03:03.332592 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.320944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38211 (local address 127.4.243.194:52611)
0925 15:03:03.322039 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:03.322084 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:03.324233 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:03.324870 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:03.324902 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:03.325026 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:03.325586 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.325602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.327108 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.327112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.328328 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.328339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.329391 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.329396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.329589 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.330298 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:03.330301 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:03.330338 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:03.330411 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:03.330841 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:03.330853 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:03.330861 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:03:03.331065 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 15:03:03.331451 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":378,"threads_started":1}
I20260925 15:03:03.333992 23070 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38211
I20260925 15:03:03.334290 23070 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:03.335027 23070 heartbeater.cc:511] Master 127.4.243.254:38211 requested a full tablet report, sending...
I20260925 15:03:03.336436 22753 ts_manager.cc:194] Registered new tserver with Master: a45196745c0b45b2b2b2899ab58799af (127.4.243.194:46561)
I20260925 15:03:03.337235 22753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.194:52611
W20260925 15:03:03.649142 23073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:03.649446 23073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:03.649603 23073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:03.656699 23073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:03.656822 23073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:03.656983 23073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:03:03.665493 23073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:38211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:03.666803 23073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:03.668489 23073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:03.681178 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:03.681197 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:03.681124 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:03.682085 23073 server_base.cc:1034] running on GCE node
I20260925 15:03:03.682713 23073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:03.683681 23073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:03.684908 23073 hybrid_clock.cc:648] HybridClock initialized: now 1790348583684862 us; error 40 us; skew 500 ppm
I20260925 15:03:03.685240 23073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:03.687587 23073 webserver.cc:460] Webserver started at http://127.4.243.195:38211/ using document root <none> and password file <none>
I20260925 15:03:03.688241 23073 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:03.688369 23073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:03.688740 23073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:03.691545 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "8674c451e9384432a558a93032ded1a7"
format_stamp: "Formatted at 2026-09-25 15:03:03 on dist-test-slave-qxh0"
server_key: "0b463ca8c947ef126b93ab6b97f3f615"
server_key_iv: "a663d74087b3ad15daa72a5e0b2e2152"
server_key_version: "encryptionkey@0"
I20260925 15:03:03.692332 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "8674c451e9384432a558a93032ded1a7"
format_stamp: "Formatted at 2026-09-25 15:03:03 on dist-test-slave-qxh0"
server_key: "0b463ca8c947ef126b93ab6b97f3f615"
server_key_iv: "a663d74087b3ad15daa72a5e0b2e2152"
server_key_version: "encryptionkey@0"
I20260925 15:03:03.697283 23073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:03:03.700555 23087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:03.701810 23073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:03:03.701974 23073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "8674c451e9384432a558a93032ded1a7"
format_stamp: "Formatted at 2026-09-25 15:03:03 on dist-test-slave-qxh0"
server_key: "0b463ca8c947ef126b93ab6b97f3f615"
server_key_iv: "a663d74087b3ad15daa72a5e0b2e2152"
server_key_version: "encryptionkey@0"
I20260925 15:03:03.702175 23073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:03.732084 23073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:03.732963 23073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:03.733299 23073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:03.734256 23073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:03.735780 23073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:03.735900 23073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:03.736030 23073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:03.736115 23073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:03.768983 23073 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:43347
I20260925 15:03:03.769068 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:43347 every 8 connection(s)
I20260925 15:03:03.770278 23073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:03:03.779850 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 23073
I20260925 15:03:03.780004 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:03:03.780519 5071 external_mini_cluster.cc:1428] Setting key 216c1682e36dc53841b98141bdd9dc3f
I20260925 15:03:03.783556 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.773142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:47533 (local address 127.4.243.254:38211)
0925 15:03:03.773366 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:03.773373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:03.774888 (+ 1515us) server_negotiation.cc:407] Connection header received
0925 15:03:03.776646 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:03.776650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:03.776729 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:03.776818 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:03.777879 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.778770 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.780384 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.780999 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.781111 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.782586 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:03.782637 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:03.782642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:03.782669 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:03.782701 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:03.782717 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:03.782814 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:03.783066 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:03:03.783182 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:03:03.784555 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.773766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:38211 (local address 127.4.243.195:47533)
0925 15:03:03.774733 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:03.774775 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:03.776363 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:03.777014 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:03.777022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:03.777106 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:03.777644 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.777663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.778931 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.778935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.780246 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.780258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.781236 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.781250 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.781444 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.782256 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:03.782269 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:03.782311 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:03.782383 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:03.782814 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:03.782827 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:03.782836 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:03:03.783024 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 15:03:03.783401 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":284,"threads_started":1}
I20260925 15:03:03.785809 23200 heartbeater.cc:347] Connected to a master server at 127.4.243.254:38211
I20260925 15:03:03.786062 23200 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:03.786819 23200 heartbeater.cc:511] Master 127.4.243.254:38211 requested a full tablet report, sending...
I20260925 15:03:03.788213 22753 ts_manager.cc:194] Registered new tserver with Master: 8674c451e9384432a558a93032ded1a7 (127.4.243.195:43347)
I20260925 15:03:03.789019 22753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.195:47533
I20260925 15:03:03.795591 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:03:03.805859 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.799756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.4.243.254:38211)
0925 15:03:03.799987 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:03.799994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:03.800373 (+ 379us) server_negotiation.cc:407] Connection header received
0925 15:03:03.800570 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:03.800574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:03.800647 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:03.800755 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:03.801238 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.801950 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.803403 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.803837 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.803978 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.805059 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:03.805063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:03.805067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:03.805085 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:03.805117 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:03.805128 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:03.805250 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:03.805430 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 15:03:03.805503 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:03:03.809721 22753 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:03:03.816627 22753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:03:03.852020 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.828996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.4.243.195:43347)
0925 15:03:03.842819 (+ 13823us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:03.842832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:03.842870 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:03:03.842968 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:03.842975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:03.843298 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:03.843902 (+ 604us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:03.845027 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.846743 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.849849 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.850805 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.850960 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.851274 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:03.851469 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:03:03.851637 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13559,"thread_start_us":14517,"threads_started":1}
I20260925 15:03:03.852074 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.828416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:43347 (local address 127.0.0.1:38516)
0925 15:03:03.840547 (+ 12131us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:03.840575 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:03.840683 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:03.844142 (+ 3459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:03.844152 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:03.844187 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:03.844740 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.844757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.846928 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.846931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.849597 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.849613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.850987 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.850993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.851212 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.851272 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:03:03.851429 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 15:03:03.851640 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12012,"mutex_wait_us":208,"spinlock_wait_cycles":1280,"thread_start_us":11885,"threads_started":1}
I20260925 15:03:03.852617 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.829503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.4.243.194:46561)
0925 15:03:03.843156 (+ 13653us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:03.843181 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:03.844238 (+ 1057us) server_negotiation.cc:407] Connection header received
0925 15:03:03.844814 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:03.844824 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:03.845035 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:03.845214 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:03.846009 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.847201 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.849805 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.851200 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.851325 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.851534 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:03.852153 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 15:03:03.852309 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13447,"thread_start_us":13314,"threads_started":1}
I20260925 15:03:03.852764 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.828910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:46561 (local address 127.0.0.1:59142)
0925 15:03:03.844126 (+ 15216us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:03.844143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:03.844283 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:03.845418 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:03.845423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:03.845444 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:03.845829 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.845842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.847351 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.847356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.849604 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.849613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.851647 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.851653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.851831 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.851859 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:03:03.851929 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:03:03.852026 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15083,"mutex_wait_us":27,"thread_start_us":9296,"threads_started":1}
I20260925 15:03:03.853200 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.828313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:36963 (local address 127.0.0.1:46022)
0925 15:03:03.839081 (+ 10768us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:03.839148 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:03.839393 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:03.844409 (+ 5016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:03.844416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:03.844444 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:03.844955 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.844968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.846891 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.846897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.849322 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:03.849337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.851514 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.851520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:03.851717 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.851750 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:03:03.851832 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:03:03.851968 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10017,"mutex_wait_us":245,"thread_start_us":15523,"threads_started":1}
I20260925 15:03:03.853775 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:03.828476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.4.243.193:36963)
0925 15:03:03.843680 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:03.843691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:03.843729 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:03:03.843851 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:03.843859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:03.844129 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:03.844426 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:03.845132 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.846741 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.849537 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:03.850963 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:03.851110 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:03.851371 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:03.852142 (+ 771us) server_negotiation.cc:299] Negotiation successful
0925 15:03:03.852314 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14991,"thread_start_us":14860,"threads_started":1}
I20260925 15:03:03.856280 23135 tablet_service.cc:1467] Processing CreateTablet for tablet a59129f5bcef4401ac2c10b2888cec56 (DEFAULT_TABLE table=test-table [id=f9627e5b1c014ce08584314dc66f6764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:03.856594 23005 tablet_service.cc:1467] Processing CreateTablet for tablet a59129f5bcef4401ac2c10b2888cec56 (DEFAULT_TABLE table=test-table [id=f9627e5b1c014ce08584314dc66f6764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:03.857049 22874 tablet_service.cc:1467] Processing CreateTablet for tablet a59129f5bcef4401ac2c10b2888cec56 (DEFAULT_TABLE table=test-table [id=f9627e5b1c014ce08584314dc66f6764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:03.857784 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59129f5bcef4401ac2c10b2888cec56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:03.858317 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59129f5bcef4401ac2c10b2888cec56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:03.858317 23005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59129f5bcef4401ac2c10b2888cec56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:03.868942 23218 tablet_bootstrap.cc:492] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Bootstrap starting.
I20260925 15:03:03.872650 23218 tablet_bootstrap.cc:654] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:03.873436 23220 tablet_bootstrap.cc:492] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Bootstrap starting.
I20260925 15:03:03.872733 23219 tablet_bootstrap.cc:492] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Bootstrap starting.
I20260925 15:03:03.874002 23218 log.cc:824] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:03.876372 23220 tablet_bootstrap.cc:654] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:03.877182 23219 tablet_bootstrap.cc:654] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:03.877584 23220 log.cc:824] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:03.878103 23218 tablet_bootstrap.cc:492] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: No bootstrap required, opened a new log
I20260925 15:03:03.878362 23219 log.cc:824] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:03.878437 23218 ts_tablet_manager.cc:1397] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260925 15:03:03.880890 23220 tablet_bootstrap.cc:492] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: No bootstrap required, opened a new log
I20260925 15:03:03.881175 23220 ts_tablet_manager.cc:1397] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 15:03:03.881750 23219 tablet_bootstrap.cc:492] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: No bootstrap required, opened a new log
I20260925 15:03:03.882009 23219 ts_tablet_manager.cc:1397] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:03:03.886390 23218 raft_consensus.cc:348] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:03.886816 23218 raft_consensus.cc:374] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:03.886945 23218 raft_consensus.cc:729] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8674c451e9384432a558a93032ded1a7, State: Initialized, Role: FOLLOWER
I20260925 15:03:03.887602 23218 consensus_queue.cc:260] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:03.888808 23200 heartbeater.cc:503] Master 127.4.243.254:38211 was elected leader, sending a full tablet report...
I20260925 15:03:03.889454 23218 ts_tablet_manager.cc:1428] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 15:03:03.893829 23220 raft_consensus.cc:348] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:03.894359 23220 raft_consensus.cc:374] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:03.894479 23220 raft_consensus.cc:729] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45196745c0b45b2b2b2899ab58799af, State: Initialized, Role: FOLLOWER
I20260925 15:03:03.895241 23220 consensus_queue.cc:260] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:03.895597 23219 raft_consensus.cc:348] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:03.896116 23219 raft_consensus.cc:374] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:03.896286 23219 raft_consensus.cc:729] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3eeea8c28e4d96b8950883f3ebdad9, State: Initialized, Role: FOLLOWER
I20260925 15:03:03.896919 23219 consensus_queue.cc:260] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:03.898118 22939 heartbeater.cc:503] Master 127.4.243.254:38211 was elected leader, sending a full tablet report...
I20260925 15:03:03.898705 23219 ts_tablet_manager.cc:1428] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260925 15:03:03.899732 23070 heartbeater.cc:503] Master 127.4.243.254:38211 was elected leader, sending a full tablet report...
I20260925 15:03:03.900273 23220 ts_tablet_manager.cc:1428] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
W20260925 15:03:04.022051 23201 tablet.cc:2369] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:03:04.069926 23071 tablet.cc:2369] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:03:04.104863 23226 raft_consensus.cc:482] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:03:04.105276 23226 raft_consensus.cc:504] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:04.107107 23226 leader_election.cc:288] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8674c451e9384432a558a93032ded1a7 (127.4.243.195:43347), a45196745c0b45b2b2b2899ab58799af (127.4.243.194:46561)
I20260925 15:03:04.113767 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.107622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:42069 (local address 127.4.243.194:46561)
0925 15:03:04.107840 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:04.107846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:04.108662 (+ 816us) server_negotiation.cc:407] Connection header received
0925 15:03:04.108871 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:04.108875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:04.108943 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:04.109050 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:04.109694 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.110364 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.112163 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.112881 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.113053 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.113122 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:04.113397 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:03:04.113520 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:03:04.113790 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.107540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:53155 (local address 127.4.243.195:43347)
0925 15:03:04.107729 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:04.107737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:04.108605 (+ 868us) server_negotiation.cc:407] Connection header received
0925 15:03:04.108827 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:04.108839 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:04.108898 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:04.108988 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:04.109565 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.110289 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.112099 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.112877 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.113075 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.113152 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:04.113429 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:03:04.113546 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:03:04.113899 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.107701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:43347 (local address 127.4.243.193:53155)
0925 15:03:04.108488 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:04.108504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:04.108682 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:04.109013 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:04.109018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:04.109039 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:04.109423 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:04.109432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.110420 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.110426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:04.111911 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:04.111922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.113132 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.113139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:04.113279 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.113300 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:03:04.113402 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:03:04.113533 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":7,"thread_start_us":440,"threads_started":1}
I20260925 15:03:04.113945 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.107741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:46561 (local address 127.4.243.193:42069)
0925 15:03:04.108546 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:04.108560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:04.108724 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:04.109063 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:04.109067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:04.109092 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:04.109533 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:04.109542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.110493 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.110499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:04.111959 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:04.111977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.113069 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.113076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:04.113242 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.113264 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:03:04.113370 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:03:04.113499 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":24,"thread_start_us":474,"threads_started":1}
I20260925 15:03:04.114926 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59129f5bcef4401ac2c10b2888cec56" candidate_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8674c451e9384432a558a93032ded1a7" is_pre_election: true
I20260925 15:03:04.115005 23025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59129f5bcef4401ac2c10b2888cec56" candidate_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45196745c0b45b2b2b2899ab58799af" is_pre_election: true
I20260925 15:03:04.115444 23155 raft_consensus.cc:2454] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3eeea8c28e4d96b8950883f3ebdad9 in term 0.
I20260925 15:03:04.115481 23025 raft_consensus.cc:2454] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3eeea8c28e4d96b8950883f3ebdad9 in term 0.
I20260925 15:03:04.116061 22827 leader_election.cc:302] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b3eeea8c28e4d96b8950883f3ebdad9, a45196745c0b45b2b2b2899ab58799af; no voters:
I20260925 15:03:04.116473 23226 raft_consensus.cc:2781] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:03:04.116641 23226 raft_consensus.cc:482] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:03:04.116761 23226 raft_consensus.cc:3037] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:04.119120 23226 raft_consensus.cc:504] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:04.119709 23226 leader_election.cc:288] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [CANDIDATE]: Term 1 election: Requested vote from peers 8674c451e9384432a558a93032ded1a7 (127.4.243.195:43347), a45196745c0b45b2b2b2899ab58799af (127.4.243.194:46561)
I20260925 15:03:04.120260 23025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59129f5bcef4401ac2c10b2888cec56" candidate_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45196745c0b45b2b2b2899ab58799af"
I20260925 15:03:04.120292 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59129f5bcef4401ac2c10b2888cec56" candidate_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8674c451e9384432a558a93032ded1a7"
I20260925 15:03:04.120500 23155 raft_consensus.cc:3037] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:04.120476 23025 raft_consensus.cc:3037] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:04.122772 23155 raft_consensus.cc:2454] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3eeea8c28e4d96b8950883f3ebdad9 in term 1.
I20260925 15:03:04.122787 23025 raft_consensus.cc:2454] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3eeea8c28e4d96b8950883f3ebdad9 in term 1.
I20260925 15:03:04.123229 22830 leader_election.cc:302] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8674c451e9384432a558a93032ded1a7, 8b3eeea8c28e4d96b8950883f3ebdad9; no voters:
I20260925 15:03:04.123561 23226 raft_consensus.cc:2781] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:04.124061 23226 raft_consensus.cc:686] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3eeea8c28e4d96b8950883f3ebdad9, State: Running, Role: LEADER
I20260925 15:03:04.124553 23226 consensus_queue.cc:237] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } }
I20260925 15:03:04.128890 22753 catalog_manager.cc:5495] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b3eeea8c28e4d96b8950883f3ebdad9 (127.4.243.193). New cstate: current_term: 1 leader_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3eeea8c28e4d96b8950883f3ebdad9" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 36963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 } health_report { overall_health: UNKNOWN } } }
W20260925 15:03:04.139005 22940 tablet.cc:2369] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:03:04.146387 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.141167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.4.243.193:36963)
0925 15:03:04.141386 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:04.141391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:04.141408 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:03:04.141572 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:04.141576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:04.141658 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:04.141779 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:04.142146 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.142911 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.144370 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.144765 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.144854 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.145323 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:04.145817 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:04.145926 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:04.146013 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:03:04.146073 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:03:04.153999 23025 raft_consensus.cc:1261] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 1 FOLLOWER]: Refusing update from remote peer 8b3eeea8c28e4d96b8950883f3ebdad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:04.153998 23155 raft_consensus.cc:1261] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 1 FOLLOWER]: Refusing update from remote peer 8b3eeea8c28e4d96b8950883f3ebdad9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:04.154913 23229 consensus_queue.cc:1035] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8674c451e9384432a558a93032ded1a7" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 43347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:04.155253 23226 consensus_queue.cc:1035] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a45196745c0b45b2b2b2899ab58799af" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:04.166329 23233 mvcc.cc:205] Tried to move back new op lower bound from 7333267800685756416 to 7333267800577720320. Current Snapshot: MvccSnapshot[applied={T|T < 7333267800685756416}]
I20260925 15:03:04.167420 23232 mvcc.cc:205] Tried to move back new op lower bound from 7333267800685756416 to 7333267800577720320. Current Snapshot: MvccSnapshot[applied={T|T < 7333267800685756416}]
I20260925 15:03:04.182436 22751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:04.182753 22751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:04.186630 22751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98cf2adfc08b4407a6aa5e1b80063615: Sending DeleteTablet for 3 replicas of tablet a59129f5bcef4401ac2c10b2888cec56
I20260925 15:03:04.187970 22874 tablet_service.cc:1514] Processing DeleteTablet for tablet a59129f5bcef4401ac2c10b2888cec56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:04 UTC) from {username='slave'} at 127.0.0.1:46022
I20260925 15:03:04.188149 23005 tablet_service.cc:1514] Processing DeleteTablet for tablet a59129f5bcef4401ac2c10b2888cec56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:04 UTC) from {username='slave'} at 127.0.0.1:59142
I20260925 15:03:04.188771 23135 tablet_service.cc:1514] Processing DeleteTablet for tablet a59129f5bcef4401ac2c10b2888cec56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:04 UTC) from {username='slave'} at 127.0.0.1:38516
I20260925 15:03:04.189252 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 22812
I20260925 15:03:04.191280 23242 tablet_replica.cc:330] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: stopping tablet replica
I20260925 15:03:04.191875 23242 raft_consensus.cc:2229] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:03:04.192741 23242 raft_consensus.cc:2258] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:04.195291 23242 ts_tablet_manager.cc:1905] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:04.195999 23244 tablet_replica.cc:330] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: stopping tablet replica
I20260925 15:03:04.196435 23244 raft_consensus.cc:2229] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:04.197010 23244 raft_consensus.cc:2258] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:04.201493 23244 ts_tablet_manager.cc:1905] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:04.203141 23243 tablet_replica.cc:330] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: stopping tablet replica
I20260925 15:03:04.203730 23243 raft_consensus.cc:2229] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:04.204319 23243 raft_consensus.cc:2258] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:04.206282 23242 ts_tablet_manager.cc:1918] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:04.206555 23242 log.cc:1196] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/a59129f5bcef4401ac2c10b2888cec56
I20260925 15:03:04.206620 23243 ts_tablet_manager.cc:1905] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:04.207314 23242 ts_tablet_manager.cc:1939] T a59129f5bcef4401ac2c10b2888cec56 P 8b3eeea8c28e4d96b8950883f3ebdad9: Deleting consensus metadata
I20260925 15:03:04.208415 23244 ts_tablet_manager.cc:1918] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:04.208583 23244 log.cc:1196] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/a59129f5bcef4401ac2c10b2888cec56
I20260925 15:03:04.209087 23244 ts_tablet_manager.cc:1939] T a59129f5bcef4401ac2c10b2888cec56 P 8674c451e9384432a558a93032ded1a7: Deleting consensus metadata
I20260925 15:03:04.209378 22738 catalog_manager.cc:4841] TS 8b3eeea8c28e4d96b8950883f3ebdad9 (127.4.243.193:36963): tablet a59129f5bcef4401ac2c10b2888cec56 (table test-table [id=f9627e5b1c014ce08584314dc66f6764]) successfully deleted
I20260925 15:03:04.210893 22740 catalog_manager.cc:4841] TS 8674c451e9384432a558a93032ded1a7 (127.4.243.195:43347): tablet a59129f5bcef4401ac2c10b2888cec56 (table test-table [id=f9627e5b1c014ce08584314dc66f6764]) successfully deleted
I20260925 15:03:04.212586 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.190330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.4.243.193:36963)
0925 15:03:04.190530 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:04.190536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:04.194032 (+ 3496us) server_negotiation.cc:407] Connection header received
0925 15:03:04.194310 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:04.194317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:04.194412 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:04.194488 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:04.195339 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.196456 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.198271 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.198981 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.199089 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.211096 (+ 12007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:04.211111 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:04.211127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:04.211761 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:04.211807 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:04.211831 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:04.211892 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:04.212137 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:03:04.212252 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:03:04.213315 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:04.213624 23243 ts_tablet_manager.cc:1918] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:04.213779 23243 log.cc:1196] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/a59129f5bcef4401ac2c10b2888cec56
I20260925 15:03:04.214323 23243 ts_tablet_manager.cc:1939] T a59129f5bcef4401ac2c10b2888cec56 P a45196745c0b45b2b2b2899ab58799af: Deleting consensus metadata
I20260925 15:03:04.216020 22737 catalog_manager.cc:4841] TS a45196745c0b45b2b2b2899ab58799af (127.4.243.194:46561): tablet a59129f5bcef4401ac2c10b2888cec56 (table test-table [id=f9627e5b1c014ce08584314dc66f6764]) successfully deleted
I20260925 15:03:04.309273 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 22812
I20260925 15:03:04.327976 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 22943
I20260925 15:03:04.335675 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.328910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.4.243.194:46561)
0925 15:03:04.329080 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:04.329084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:04.329102 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:03:04.329262 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:04.329266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:04.329371 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:04.329456 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:04.329913 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.330801 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.332614 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.333054 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.333155 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.334225 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:04.334234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:04.334247 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:04.334936 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:04.334981 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:04.335001 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:04.335092 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:04.335255 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:03:04.335338 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:03:04.336442 23065 generic_service.cc:196] Checking for leaks (request via RPC)
W20260925 15:03:04.345219 22784 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.4.243.254:38211
I20260925 15:03:04.428542 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 22943
I20260925 15:03:04.445585 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 23073
I20260925 15:03:04.452582 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:04.446419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.4.243.195:43347)
0925 15:03:04.446626 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:04.446634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:04.446657 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:03:04.446895 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:04.446900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:04.446985 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:04.447110 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:04.447489 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.448207 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.449464 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:04.449872 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:04.450009 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:04.451258 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:04.451267 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:04.451284 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:04.451898 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:04.451951 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:04.451973 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:04.452081 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:04.452208 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:03:04.452304 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:03:04.453238 23195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:04.540853 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 23073
I20260925 15:03:04.557214 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 22722
I20260925 15:03:04.558357 22783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:04.667007 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 22722
2026-09-25T15:03:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:04 dist-test-slave-qxh0 krb5kdc[23252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:04 dist-test-slave-qxh0 krb5kdc[23252](info): set up 2 sockets
Sep 25 15:03:04 dist-test-slave-qxh0 krb5kdc[23252](info): commencing operation
krb5kdc: starting...
W20260925 15:03:05.766952 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:05 dist-test-slave-qxh0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:06.022884 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:34075
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:34075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:06.344726 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:06.345108 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:06.345299 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:06.352420 23265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:06.352582 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:06.352698 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:06.352802 23265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:06.352872 23265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:06.362797 23265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:34075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:34075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:06.364259 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:06.365638 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:06.377496 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:06.377542 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:06.377487 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:06.378048 23265 server_base.cc:1034] running on GCE node
I20260925 15:03:06.378983 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:06.380308 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:06.381538 23265 hybrid_clock.cc:648] HybridClock initialized: now 1790348586381493 us; error 57 us; skew 500 ppm
I20260925 15:03:06.381893 23265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:06 dist-test-slave-qxh0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:06.384914 23265 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:06.386305 23265 webserver.cc:460] Webserver started at http://127.4.243.254:43739/ using document root <none> and password file <none>
I20260925 15:03:06.387039 23265 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:06.387176 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:06.387490 23265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:06.390374 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "0c937f57713a4d08978146462e093f78"
format_stamp: "Formatted at 2026-09-25 15:03:06 on dist-test-slave-qxh0"
server_key: "78636789308c89e5e1c9dde13dfde962"
server_key_iv: "485d7a3c650884f681ea056565f969b5"
server_key_version: "encryptionkey@0"
I20260925 15:03:06.391165 23265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "0c937f57713a4d08978146462e093f78"
format_stamp: "Formatted at 2026-09-25 15:03:06 on dist-test-slave-qxh0"
server_key: "78636789308c89e5e1c9dde13dfde962"
server_key_iv: "485d7a3c650884f681ea056565f969b5"
server_key_version: "encryptionkey@0"
I20260925 15:03:06.396148 23265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:03:06.399664 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:06.401237 23265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:03:06.401435 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "0c937f57713a4d08978146462e093f78"
format_stamp: "Formatted at 2026-09-25 15:03:06 on dist-test-slave-qxh0"
server_key: "78636789308c89e5e1c9dde13dfde962"
server_key_iv: "485d7a3c650884f681ea056565f969b5"
server_key_version: "encryptionkey@0"
I20260925 15:03:06.401646 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:06.422860 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:06.427016 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:06.427307 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:06.445922 23265 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:34075
I20260925 15:03:06.445932 23331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:34075 every 8 connection(s)
I20260925 15:03:06.447525 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:06.451386 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:06.455905 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 23265
I20260925 15:03:06.456092 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:06.456528 5071 external_mini_cluster.cc:1428] Setting key 52494da31aa6a3cfcbe3f7cb17d7c348
I20260925 15:03:06.462366 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78: Bootstrap starting.
I20260925 15:03:06.465433 23332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:06.466429 23332 log.cc:824] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:06.469379 23332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78: No bootstrap required, opened a new log
Sep 25 15:03:06 dist-test-slave-qxh0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:06.477586 23332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c937f57713a4d08978146462e093f78" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34075 } }
I20260925 15:03:06.477998 23332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:06.478137 23332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c937f57713a4d08978146462e093f78, State: Initialized, Role: FOLLOWER
I20260925 15:03:06.478902 23332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c937f57713a4d08978146462e093f78" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34075 } }
I20260925 15:03:06.479187 23332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:06.479353 23332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:06.479548 23332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:06.479978 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:06.458339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.4.243.254:34075)
0925 15:03:06.459286 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:06.459304 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:06.459628 (+ 324us) server_negotiation.cc:407] Connection header received
0925 15:03:06.461453 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:06.461475 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:06.461860 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:06.462242 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:06.463801 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:06.465067 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:06.466455 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:06.467236 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:06.467484 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:06.472030 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:06.472040 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:06.472059 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:06.472098 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:06.475442 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:06.476224 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:06.476231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:06.476241 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:06.476436 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:06.476857 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:06.476861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:06.476863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:06.477383 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:06.477715 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:06.478185 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 15:03:06.478737 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":447,"threads_started":1}
I20260925 15:03:06.482018 23332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c937f57713a4d08978146462e093f78" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34075 } }
I20260925 15:03:06.482575 23332 leader_election.cc:302] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c937f57713a4d08978146462e093f78; no voters:
I20260925 15:03:06.483330 23332 leader_election.cc:288] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:06.483474 23337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:06.484169 23337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [term 1 LEADER]: Becoming Leader. State: Replica: 0c937f57713a4d08978146462e093f78, State: Running, Role: LEADER
I20260925 15:03:06.484719 23337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c937f57713a4d08978146462e093f78" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34075 } }
I20260925 15:03:06.485443 23332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:06.487567 23338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c937f57713a4d08978146462e093f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c937f57713a4d08978146462e093f78" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34075 } } }
I20260925 15:03:06.488029 23338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:06.487984 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c937f57713a4d08978146462e093f78. Latest consensus state: current_term: 1 leader_uuid: "0c937f57713a4d08978146462e093f78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c937f57713a4d08978146462e093f78" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34075 } } }
I20260925 15:03:06.488292 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:06.490264 23346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:06.494881 23346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:06.502526 23346 catalog_manager.cc:1347] Generated new cluster ID: 1a92d53d1d8542bb92b6cc4bc93df9f5
I20260925 15:03:06.502665 23346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:06.534487 23346 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:06.536023 23346 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:06.547046 23346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78: Generated new TSK 0
I20260925 15:03:06.548126 23346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 15:03:06.613269 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:34075
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:06.940379 23358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:06.940637 23358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:06.940790 23358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:06.948110 23358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:06.948217 23358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:06.948395 23358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:03:06.958055 23358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:34075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:06.959372 23358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:06.960716 23358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:06.973891 23364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:06.974047 23366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:06.974123 23363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:06.975337 23358 server_base.cc:1034] running on GCE node
I20260925 15:03:06.975972 23358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:06.976857 23358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:06.978073 23358 hybrid_clock.cc:648] HybridClock initialized: now 1790348586978035 us; error 28 us; skew 500 ppm
I20260925 15:03:06.978425 23358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:06 dist-test-slave-qxh0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:06.981876 23358 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:03:06.983305 23358 webserver.cc:460] Webserver started at http://127.4.243.193:36261/ using document root <none> and password file <none>
I20260925 15:03:06.983996 23358 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:06.984126 23358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:06.984450 23358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:06.987516 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "aa0b23d2e13e40008b2f8ff968a974f1"
format_stamp: "Formatted at 2026-09-25 15:03:06 on dist-test-slave-qxh0"
server_key: "0a0226b2e3025aff888c6e376d4c291a"
server_key_iv: "d55ea7bc8edbde3eac3f8b94801d5fec"
server_key_version: "encryptionkey@0"
I20260925 15:03:06.988299 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "aa0b23d2e13e40008b2f8ff968a974f1"
format_stamp: "Formatted at 2026-09-25 15:03:06 on dist-test-slave-qxh0"
server_key: "0a0226b2e3025aff888c6e376d4c291a"
server_key_iv: "d55ea7bc8edbde3eac3f8b94801d5fec"
server_key_version: "encryptionkey@0"
I20260925 15:03:06.993407 23358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:03:06.996938 23373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:06.998211 23358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:03:06.998373 23358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "aa0b23d2e13e40008b2f8ff968a974f1"
format_stamp: "Formatted at 2026-09-25 15:03:06 on dist-test-slave-qxh0"
server_key: "0a0226b2e3025aff888c6e376d4c291a"
server_key_iv: "d55ea7bc8edbde3eac3f8b94801d5fec"
server_key_version: "encryptionkey@0"
I20260925 15:03:06.998559 23358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:07.018865 23358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:07.022737 23358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:07.023037 23358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:07.024092 23358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:07.025365 23358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:07.025437 23358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:07.025605 23358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:07.025665 23358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:07.058872 23358 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:39079
I20260925 15:03:07.058916 23485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:39079 every 8 connection(s)
I20260925 15:03:07.060171 23358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:03:07.067659 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 23358
I20260925 15:03:07.067823 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:03:07.068276 5071 external_mini_cluster.cc:1428] Setting key 20280c98c92870d5a2a6441d47660330
Sep 25 15:03:07 dist-test-slave-qxh0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:07.081265 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:07.062659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:53441 (local address 127.4.243.254:34075)
0925 15:03:07.063128 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:07.063132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:07.063931 (+ 799us) server_negotiation.cc:407] Connection header received
0925 15:03:07.065761 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:07.065767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:07.065859 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:07.065982 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:07.067559 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.068444 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.070019 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.070480 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.070578 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:07.075033 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:07.075038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:07.075041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:07.075071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:07.077562 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:07.078497 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:07.078502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:07.078504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:07.078675 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:07.079209 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:07.079214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:07.079217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:07.079470 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:07.079580 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:07.080464 (+ 884us) server_negotiation.cc:299] Negotiation successful
0925 15:03:07.080767 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":259,"threads_started":1}
I20260925 15:03:07.081913 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:07.063117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:34075 (local address 127.4.243.193:53441)
0925 15:03:07.063770 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:07.063806 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:07.065486 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:07.066208 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:07.066215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:07.066779 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:07.067365 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:07.067401 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.068604 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.068609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:07.069888 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:07.069903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.070731 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.070738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:07.071051 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:07.071784 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:07.071818 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:07.074816 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:07.077702 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:07.077713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:07.077733 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:07.078366 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:07.078809 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:07.078817 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:07.078822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:07.079096 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:07.079613 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:07.079623 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:07.080074 (+ 451us) client_negotiation.cc:769] Sending connection context
0925 15:03:07.080310 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 15:03:07.080715 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
I20260925 15:03:07.083633 23486 heartbeater.cc:347] Connected to a master server at 127.4.243.254:34075
I20260925 15:03:07.083974 23486 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:07.084846 23486 heartbeater.cc:511] Master 127.4.243.254:34075 requested a full tablet report, sending...
I20260925 15:03:07.087283 23297 ts_manager.cc:194] Registered new tserver with Master: aa0b23d2e13e40008b2f8ff968a974f1 (127.4.243.193:39079)
I20260925 15:03:07.088781 23297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:53441
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 15:03:07.124382 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:34075
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:07.448514 23492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:07.448788 23492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:07.448954 23492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:07.456204 23492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:07.456312 23492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:07.456489 23492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:03:07.466141 23492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:34075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:07.467466 23492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:07.468739 23492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:07.481858 23500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:07.481828 23498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:07.482219 23497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:07.482648 23492 server_base.cc:1034] running on GCE node
I20260925 15:03:07.483309 23492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:07.484268 23492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:07.485458 23492 hybrid_clock.cc:648] HybridClock initialized: now 1790348587485401 us; error 57 us; skew 500 ppm
I20260925 15:03:07.485801 23492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:07 dist-test-slave-qxh0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348587, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:07.494577 23492 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:03:07.496332 23492 webserver.cc:460] Webserver started at http://127.4.243.194:46435/ using document root <none> and password file <none>
I20260925 15:03:07.497042 23492 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:07.497184 23492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:07.497500 23492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:07.500403 23492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "b2e06787690c467fa7c3b2ecedb286cc"
format_stamp: "Formatted at 2026-09-25 15:03:07 on dist-test-slave-qxh0"
server_key: "4718c0986981fc1ef56ddb9539654be4"
server_key_iv: "1e6cbd7c46359b79cfe04c5ea61101ad"
server_key_version: "encryptionkey@0"
I20260925 15:03:07.501160 23492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "b2e06787690c467fa7c3b2ecedb286cc"
format_stamp: "Formatted at 2026-09-25 15:03:07 on dist-test-slave-qxh0"
server_key: "4718c0986981fc1ef56ddb9539654be4"
server_key_iv: "1e6cbd7c46359b79cfe04c5ea61101ad"
server_key_version: "encryptionkey@0"
I20260925 15:03:07.506455 23492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:03:07.509672 23507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:07.511035 23492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:07.511193 23492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "b2e06787690c467fa7c3b2ecedb286cc"
format_stamp: "Formatted at 2026-09-25 15:03:07 on dist-test-slave-qxh0"
server_key: "4718c0986981fc1ef56ddb9539654be4"
server_key_iv: "1e6cbd7c46359b79cfe04c5ea61101ad"
server_key_version: "encryptionkey@0"
I20260925 15:03:07.511394 23492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:07.533131 23492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:07.537217 23492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:07.537528 23492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:07.538471 23492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:07.539795 23492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:07.539899 23492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:07.540047 23492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:07.540117 23492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:07.572745 23492 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:41717
I20260925 15:03:07.572830 23619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:41717 every 8 connection(s)
I20260925 15:03:07.574041 23492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:03:07.578039 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 23492
I20260925 15:03:07.578225 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:03:07.578853 5071 external_mini_cluster.cc:1428] Setting key 6d32eab243abd634df47f1bf134f61ce
Sep 25 15:03:07 dist-test-slave-qxh0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348587, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:07.604216 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:07.576703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:45429 (local address 127.4.243.254:34075)
0925 15:03:07.576900 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:07.576906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:07.578088 (+ 1182us) server_negotiation.cc:407] Connection header received
0925 15:03:07.580968 (+ 2880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:07.580975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:07.581068 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:07.581147 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:07.583628 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.584795 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.587380 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.588143 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.588271 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:07.593094 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:07.593108 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:07.593113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:07.593200 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:07.598116 (+ 4916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:07.599316 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:07.599322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:07.599325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:07.599547 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:07.600159 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:07.600164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:07.600168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:07.600646 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:07.600888 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:07.603598 (+ 2710us) server_negotiation.cc:299] Negotiation successful
0925 15:03:07.603864 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:03:07.605352 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:07.577104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:34075 (local address 127.4.243.194:45429)
0925 15:03:07.577893 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:07.577944 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:07.580606 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:07.581709 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:07.581721 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:07.582512 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:07.583383 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:07.583413 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.584976 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.584984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:07.587207 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:07.587224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:07.588607 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:07.588618 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:07.588955 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:07.589748 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:07.589777 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:07.592571 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:07.598329 (+ 5758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:07.598340 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:07.598362 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:07.599129 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:07.599702 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:07.599710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:07.599716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:07.599982 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:07.602688 (+ 2706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:07.602696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:07.603124 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 15:03:07.603422 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 15:03:07.604086 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":258,"threads_started":1}
I20260925 15:03:07.606967 23620 heartbeater.cc:347] Connected to a master server at 127.4.243.254:34075
I20260925 15:03:07.607291 23620 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:07.607985 23620 heartbeater.cc:511] Master 127.4.243.254:34075 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 15:03:07.609561 23297 ts_manager.cc:194] Registered new tserver with Master: b2e06787690c467fa7c3b2ecedb286cc (127.4.243.194:41717)
I20260925 15:03:07.610415 23297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:45429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 15:03:07.639782 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:34075
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:07.969419 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:07.969720 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:07.969890 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:07.977370 23625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:07.977468 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:07.977686 23625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:03:07.987310 23625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:34075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:07.988636 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:07.990001 23625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:08.003144 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:08.003060 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:08.003695 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:08.004613 23625 server_base.cc:1034] running on GCE node
I20260925 15:03:08.005357 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:08.006439 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:08.007650 23625 hybrid_clock.cc:648] HybridClock initialized: now 1790348588007596 us; error 42 us; skew 500 ppm
I20260925 15:03:08.007982 23625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:08 dist-test-slave-qxh0 krb5kdc[23252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:08.011119 23625 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:03:08.012513 23625 webserver.cc:460] Webserver started at http://127.4.243.195:32865/ using document root <none> and password file <none>
I20260925 15:03:08.013171 23625 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:08.013319 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:08.013618 23625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:08.016624 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "d4060fc0b95844338fa1d9855a8c9f72"
format_stamp: "Formatted at 2026-09-25 15:03:08 on dist-test-slave-qxh0"
server_key: "22bc7e5daab27153a553c24abb155195"
server_key_iv: "f33f25c1a44704eea3330cbd4b2716c6"
server_key_version: "encryptionkey@0"
I20260925 15:03:08.017405 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "d4060fc0b95844338fa1d9855a8c9f72"
format_stamp: "Formatted at 2026-09-25 15:03:08 on dist-test-slave-qxh0"
server_key: "22bc7e5daab27153a553c24abb155195"
server_key_iv: "f33f25c1a44704eea3330cbd4b2716c6"
server_key_version: "encryptionkey@0"
I20260925 15:03:08.022423 23625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:03:08.025830 23640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:08.027169 23625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:08.027354 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "d4060fc0b95844338fa1d9855a8c9f72"
format_stamp: "Formatted at 2026-09-25 15:03:08 on dist-test-slave-qxh0"
server_key: "22bc7e5daab27153a553c24abb155195"
server_key_iv: "f33f25c1a44704eea3330cbd4b2716c6"
server_key_version: "encryptionkey@0"
I20260925 15:03:08.027550 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:08.045059 23625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:08.049226 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:08.049510 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:08.050448 23625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:08.051798 23625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:08.051900 23625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:08.052047 23625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:08.052119 23625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:08.085270 23625 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:36853
I20260925 15:03:08.085320 23752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:36853 every 8 connection(s)
I20260925 15:03:08.086889 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:03:08.092583 23486 heartbeater.cc:503] Master 127.4.243.254:34075 was elected leader, sending a full tablet report...
I20260925 15:03:08.094321 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 23625
I20260925 15:03:08.094499 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:03:08.095060 5071 external_mini_cluster.cc:1428] Setting key 0896547780985b798f79e860913f7bbf
Sep 25 15:03:08 dist-test-slave-qxh0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:08.107141 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.089541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:44177 (local address 127.4.243.254:34075)
0925 15:03:08.089760 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.089766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.091029 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 15:03:08.092901 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.092907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.093008 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.093125 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:08.094890 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.095825 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.097747 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.098168 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.098272 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.101571 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:08.101575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:08.101579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:08.101611 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:08.104320 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.104976 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.104979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.104982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.105123 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.105483 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.105486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.105488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.105769 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:08.105879 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.106650 (+ 771us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.106889 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:03:08.108006 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.090056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:34075 (local address 127.4.243.195:44177)
0925 15:03:08.090869 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.090909 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:08.092580 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:08.093357 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:08.093365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:08.093862 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:08.094725 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.094750 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.095982 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.095986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.097618 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.097629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.098357 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.098361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.098551 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.099271 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:08.099292 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:08.101362 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:08.104444 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:08.104450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:08.104462 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:08.104873 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:08.105219 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:08.105222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:08.105225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:08.105391 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:08.105858 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:08.105863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:08.106236 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 15:03:08.106464 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 15:03:08.106903 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":329,"threads_started":1}
I20260925 15:03:08.109319 23753 heartbeater.cc:347] Connected to a master server at 127.4.243.254:34075
I20260925 15:03:08.109638 23753 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:08.110402 23753 heartbeater.cc:511] Master 127.4.243.254:34075 requested a full tablet report, sending...
I20260925 15:03:08.111828 23297 ts_manager.cc:194] Registered new tserver with Master: d4060fc0b95844338fa1d9855a8c9f72 (127.4.243.195:36853)
I20260925 15:03:08.112624 23297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:44177
I20260925 15:03:08.122254 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:03:08.139012 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.126657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.4.243.254:34075)
0925 15:03:08.126834 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.126839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.127279 (+ 440us) server_negotiation.cc:407] Connection header received
0925 15:03:08.127414 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.127418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.127479 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.127562 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:08.128923 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.129623 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.131028 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.131508 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.131651 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.134121 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:08.134125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:08.134128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:08.134157 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:08.136498 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.137259 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.137263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.137265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.137389 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.137782 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.137785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.137787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.138078 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:08.138279 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.138559 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.138779 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:03:08.142993 23297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:03:08.149853 23297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:03:08.172729 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.162626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.4.243.195:36853)
0925 15:03:08.163365 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.163375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.163741 (+ 366us) server_negotiation.cc:407] Connection header received
0925 15:03:08.163987 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.163994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.164261 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.164462 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:08.165988 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.167732 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.170629 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.171539 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.171688 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.171932 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.172210 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.172374 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":323,"threads_started":1}
I20260925 15:03:08.172842 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.162582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:36853 (local address 127.0.0.1:42078)
0925 15:03:08.163596 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.163628 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:08.163790 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:08.164506 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:08.164515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:08.164551 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:08.165111 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.165129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.167868 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.167876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.170421 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.170436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.171745 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.171751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.171958 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.172008 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:03:08.172169 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 15:03:08.172364 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":660,"threads_started":1}
I20260925 15:03:08.175446 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.164722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:41717 (local address 127.0.0.1:50032)
0925 15:03:08.165371 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.165389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:08.165505 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:08.168179 (+ 2674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:08.168187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:08.168216 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:08.168719 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.168731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.170614 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.170619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.172667 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.172679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.174512 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.174519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.174814 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.174856 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:03:08.174950 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:03:08.175072 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":27,"thread_start_us":686,"threads_started":1}
I20260925 15:03:08.176127 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.165274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.4.243.194:41717)
0925 15:03:08.167457 (+ 2183us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.167468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.167503 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:03:08.167627 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.167637 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.167946 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.168553 (+ 607us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:08.168901 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.170457 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.172887 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.174299 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.174430 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.174625 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.175682 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.175858 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"thread_start_us":1870,"threads_started":1}
I20260925 15:03:08.177156 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.165157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50472)
0925 15:03:08.168351 (+ 3194us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.168372 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:08.168916 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:08.169890 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:08.169897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:08.169922 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:08.170415 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.170429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.172393 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.172398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.174815 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.174827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.176325 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.176336 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.176521 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.176562 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:03:08.176649 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:03:08.176801 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3027,"mutex_wait_us":30,"thread_start_us":2872,"threads_started":1}
I20260925 15:03:08.177649 23688 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9f1128725d4067b0fda94fbad75d7e (DEFAULT_TABLE table=test-table [id=f085eb6e9c33413ca65bb5bca55a00d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:08.179054 23688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9f1128725d4067b0fda94fbad75d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:08.179833 23555 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9f1128725d4067b0fda94fbad75d7e (DEFAULT_TABLE table=test-table [id=f085eb6e9c33413ca65bb5bca55a00d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:08.180954 23555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9f1128725d4067b0fda94fbad75d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:08.183068 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.165744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.4.243.193:39079)
0925 15:03:08.166965 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.166977 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.168479 (+ 1502us) server_negotiation.cc:407] Connection header received
0925 15:03:08.169256 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.169263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.169499 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.169667 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:08.170627 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.172219 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.174962 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.176009 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.176154 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.176377 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.177053 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.177233 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":833,"threads_started":1}
I20260925 15:03:08.186846 23421 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9f1128725d4067b0fda94fbad75d7e (DEFAULT_TABLE table=test-table [id=f085eb6e9c33413ca65bb5bca55a00d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:08.188257 23421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9f1128725d4067b0fda94fbad75d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:08.195323 23772 tablet_bootstrap.cc:492] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Bootstrap starting.
I20260925 15:03:08.198091 23771 tablet_bootstrap.cc:492] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Bootstrap starting.
I20260925 15:03:08.198957 23772 tablet_bootstrap.cc:654] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:08.200215 23772 log.cc:824] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:08.201689 23771 tablet_bootstrap.cc:654] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:08.203239 23771 log.cc:824] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:08.203465 23774 tablet_bootstrap.cc:492] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Bootstrap starting.
I20260925 15:03:08.204370 23772 tablet_bootstrap.cc:492] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: No bootstrap required, opened a new log
I20260925 15:03:08.204713 23772 ts_tablet_manager.cc:1397] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 15:03:08.207046 23774 tablet_bootstrap.cc:654] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:08.208124 23774 log.cc:824] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:08.209302 23771 tablet_bootstrap.cc:492] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: No bootstrap required, opened a new log
I20260925 15:03:08.209570 23771 ts_tablet_manager.cc:1397] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260925 15:03:08.210511 23774 tablet_bootstrap.cc:492] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: No bootstrap required, opened a new log
I20260925 15:03:08.210780 23774 ts_tablet_manager.cc:1397] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:03:08.215591 23772 raft_consensus.cc:348] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.216058 23772 raft_consensus.cc:374] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:08.216238 23772 raft_consensus.cc:729] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e06787690c467fa7c3b2ecedb286cc, State: Initialized, Role: FOLLOWER
I20260925 15:03:08.216833 23772 consensus_queue.cc:260] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.217980 23620 heartbeater.cc:503] Master 127.4.243.254:34075 was elected leader, sending a full tablet report...
I20260925 15:03:08.218962 23774 raft_consensus.cc:348] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.219455 23774 raft_consensus.cc:374] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:08.219573 23772 ts_tablet_manager.cc:1428] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20260925 15:03:08.219599 23774 raft_consensus.cc:729] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0b23d2e13e40008b2f8ff968a974f1, State: Initialized, Role: FOLLOWER
I20260925 15:03:08.220000 23771 raft_consensus.cc:348] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.220237 23774 consensus_queue.cc:260] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.220451 23771 raft_consensus.cc:374] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:08.220539 23771 raft_consensus.cc:729] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4060fc0b95844338fa1d9855a8c9f72, State: Initialized, Role: FOLLOWER
I20260925 15:03:08.221115 23771 consensus_queue.cc:260] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.222335 23753 heartbeater.cc:503] Master 127.4.243.254:34075 was elected leader, sending a full tablet report...
I20260925 15:03:08.222854 23771 ts_tablet_manager.cc:1428] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260925 15:03:08.223287 23774 ts_tablet_manager.cc:1428] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:03:08.233029 23777 raft_consensus.cc:482] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:03:08.233253 23777 raft_consensus.cc:504] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.234634 23777 leader_election.cc:288] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4060fc0b95844338fa1d9855a8c9f72 (127.4.243.195:36853), aa0b23d2e13e40008b2f8ff968a974f1 (127.4.243.193:39079)
I20260925 15:03:08.241194 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.235067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:41111 (local address 127.4.243.195:36853)
0925 15:03:08.235286 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.235294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.235520 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:03:08.235776 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.235783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.235861 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.235927 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:08.236692 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.237554 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.239386 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.240182 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.240345 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.240398 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.240802 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.240953 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:03:08.241227 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.234828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:36853 (local address 127.4.243.194:41111)
0925 15:03:08.235381 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.235425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:08.235574 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:08.236101 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:08.236104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:08.236150 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:08.236542 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.236556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.237670 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.237676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.239239 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.239250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.240421 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.240430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.240614 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.240662 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:03:08.240748 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:03:08.240890 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":289,"threads_started":1}
I20260925 15:03:08.241305 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.235722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:36603 (local address 127.4.243.193:39079)
0925 15:03:08.235901 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.235908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.236163 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:03:08.236277 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.236284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.236351 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.236460 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:08.237051 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.237808 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.239537 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.240373 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.240530 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.240606 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.240941 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.241080 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:03:08.242026 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.235465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.4.243.194:36603)
0925 15:03:08.236064 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.236077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:08.236175 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:08.236473 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:08.236478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:08.236494 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:08.236890 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.236902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.237937 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.237940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.239399 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:08.239411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.240580 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.240584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:08.240754 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.240794 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:03:08.241476 (+ 682us) client_negotiation.cc:240] Negotiation successful
0925 15:03:08.241602 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":3,"thread_start_us":322,"threads_started":1}
I20260925 15:03:08.242751 23708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f1128725d4067b0fda94fbad75d7e" candidate_uuid: "b2e06787690c467fa7c3b2ecedb286cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4060fc0b95844338fa1d9855a8c9f72" is_pre_election: true
I20260925 15:03:08.243098 23441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f1128725d4067b0fda94fbad75d7e" candidate_uuid: "b2e06787690c467fa7c3b2ecedb286cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" is_pre_election: true
I20260925 15:03:08.243304 23708 raft_consensus.cc:2454] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e06787690c467fa7c3b2ecedb286cc in term 0.
I20260925 15:03:08.243613 23441 raft_consensus.cc:2454] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e06787690c467fa7c3b2ecedb286cc in term 0.
I20260925 15:03:08.244104 23509 leader_election.cc:302] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e06787690c467fa7c3b2ecedb286cc, d4060fc0b95844338fa1d9855a8c9f72; no voters:
I20260925 15:03:08.244536 23777 raft_consensus.cc:2781] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:03:08.244686 23777 raft_consensus.cc:482] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:03:08.244818 23777 raft_consensus.cc:3037] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:08.247104 23777 raft_consensus.cc:504] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.247681 23777 leader_election.cc:288] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [CANDIDATE]: Term 1 election: Requested vote from peers d4060fc0b95844338fa1d9855a8c9f72 (127.4.243.195:36853), aa0b23d2e13e40008b2f8ff968a974f1 (127.4.243.193:39079)
I20260925 15:03:08.248085 23708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f1128725d4067b0fda94fbad75d7e" candidate_uuid: "b2e06787690c467fa7c3b2ecedb286cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4060fc0b95844338fa1d9855a8c9f72"
I20260925 15:03:08.248163 23441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9f1128725d4067b0fda94fbad75d7e" candidate_uuid: "b2e06787690c467fa7c3b2ecedb286cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0b23d2e13e40008b2f8ff968a974f1"
I20260925 15:03:08.248289 23708 raft_consensus.cc:3037] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:08.248369 23441 raft_consensus.cc:3037] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:08.250448 23708 raft_consensus.cc:2454] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e06787690c467fa7c3b2ecedb286cc in term 1.
I20260925 15:03:08.250509 23441 raft_consensus.cc:2454] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e06787690c467fa7c3b2ecedb286cc in term 1.
I20260925 15:03:08.250842 23509 leader_election.cc:302] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e06787690c467fa7c3b2ecedb286cc, d4060fc0b95844338fa1d9855a8c9f72; no voters:
I20260925 15:03:08.251117 23777 raft_consensus.cc:2781] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:08.251547 23777 raft_consensus.cc:686] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 1 LEADER]: Becoming Leader. State: Replica: b2e06787690c467fa7c3b2ecedb286cc, State: Running, Role: LEADER
I20260925 15:03:08.252126 23777 consensus_queue.cc:237] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } }
I20260925 15:03:08.256597 23297 catalog_manager.cc:5495] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc reported cstate change: term changed from 0 to 1, leader changed from <none> to b2e06787690c467fa7c3b2ecedb286cc (127.4.243.194). New cstate: current_term: 1 leader_uuid: "b2e06787690c467fa7c3b2ecedb286cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e06787690c467fa7c3b2ecedb286cc" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 41717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 } health_report { overall_health: UNKNOWN } } }
I20260925 15:03:08.272661 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.267492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.4.243.194:41717)
0925 15:03:08.267715 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.267719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.267736 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:03:08.267892 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.267896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.267966 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.268079 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:08.268487 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.269222 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.270505 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.270971 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.271063 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.271571 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:08.272020 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:08.272147 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.272313 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.272377 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 15:03:08.280762 23708 raft_consensus.cc:1261] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 1 FOLLOWER]: Refusing update from remote peer b2e06787690c467fa7c3b2ecedb286cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:08.280761 23441 raft_consensus.cc:1261] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 1 FOLLOWER]: Refusing update from remote peer b2e06787690c467fa7c3b2ecedb286cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:08.281771 23782 consensus_queue.cc:1035] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa0b23d2e13e40008b2f8ff968a974f1" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:08.282192 23777 consensus_queue.cc:1035] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4060fc0b95844338fa1d9855a8c9f72" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:08.295128 23785 mvcc.cc:205] Tried to move back new op lower bound from 7333267817587671040 to 7333267817484812288. Current Snapshot: MvccSnapshot[applied={T|T < 7333267817587671040}]
I20260925 15:03:08.300520 23787 mvcc.cc:205] Tried to move back new op lower bound from 7333267817587671040 to 7333267817484812288. Current Snapshot: MvccSnapshot[applied={T|T < 7333267817587671040}]
I20260925 15:03:08.309029 23295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:08.309270 23295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:08.313172 23295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c937f57713a4d08978146462e093f78: Sending DeleteTablet for 3 replicas of tablet 4f9f1128725d4067b0fda94fbad75d7e
I20260925 15:03:08.314342 23555 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f9f1128725d4067b0fda94fbad75d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:08 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:50032
I20260925 15:03:08.314440 23421 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f9f1128725d4067b0fda94fbad75d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:08 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:50472
I20260925 15:03:08.315084 23688 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f9f1128725d4067b0fda94fbad75d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:08 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:42078
I20260925 15:03:08.318023 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 23358
I20260925 15:03:08.323408 23795 tablet_replica.cc:330] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: stopping tablet replica
I20260925 15:03:08.323978 23795 raft_consensus.cc:2229] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:08.324826 23795 raft_consensus.cc:2258] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:08.325436 23796 tablet_replica.cc:330] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: stopping tablet replica
I20260925 15:03:08.328783 23795 ts_tablet_manager.cc:1905] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:08.328989 23796 raft_consensus.cc:2229] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:08.329496 23796 raft_consensus.cc:2258] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:08.331354 23794 tablet_replica.cc:330] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: stopping tablet replica
I20260925 15:03:08.331769 23794 raft_consensus.cc:2229] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:03:08.332582 23794 raft_consensus.cc:2258] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:08.333063 23796 ts_tablet_manager.cc:1905] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:08.334656 23794 ts_tablet_manager.cc:1905] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:03:08 dist-test-slave-qxh0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 15:03:08.342016 23794 ts_tablet_manager.cc:1918] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:08.342294 23794 log.cc:1196] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/4f9f1128725d4067b0fda94fbad75d7e
I20260925 15:03:08.342985 23794 ts_tablet_manager.cc:1939] T 4f9f1128725d4067b0fda94fbad75d7e P b2e06787690c467fa7c3b2ecedb286cc: Deleting consensus metadata
I20260925 15:03:08.344532 23795 ts_tablet_manager.cc:1918] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:08.344722 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.319521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.4.243.193:39079)
0925 15:03:08.319705 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.319711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.322961 (+ 3250us) server_negotiation.cc:407] Connection header received
0925 15:03:08.323841 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.323847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.323940 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.324022 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:08.327712 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.328871 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.330676 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.331373 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.331517 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.337082 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:08.337095 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:08.337112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:08.337163 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:08.341133 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.342074 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.342084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.342088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.342255 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.342780 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.342786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.342789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.343232 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:08.343529 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.344105 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.344338 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:03:08.344893 23281 catalog_manager.cc:4841] TS b2e06787690c467fa7c3b2ecedb286cc (127.4.243.194:41717): tablet 4f9f1128725d4067b0fda94fbad75d7e (table test-table [id=f085eb6e9c33413ca65bb5bca55a00d4]) successfully deleted
I20260925 15:03:08.346583 23481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:08.346686 23796 ts_tablet_manager.cc:1918] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:08.346858 23796 log.cc:1196] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/4f9f1128725d4067b0fda94fbad75d7e
I20260925 15:03:08.347074 23795 log.cc:1196] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/4f9f1128725d4067b0fda94fbad75d7e
I20260925 15:03:08.347703 23795 ts_tablet_manager.cc:1939] T 4f9f1128725d4067b0fda94fbad75d7e P d4060fc0b95844338fa1d9855a8c9f72: Deleting consensus metadata
I20260925 15:03:08.349897 23282 catalog_manager.cc:4841] TS d4060fc0b95844338fa1d9855a8c9f72 (127.4.243.195:36853): tablet 4f9f1128725d4067b0fda94fbad75d7e (table test-table [id=f085eb6e9c33413ca65bb5bca55a00d4]) successfully deleted
I20260925 15:03:08.440734 23796 ts_tablet_manager.cc:1939] T 4f9f1128725d4067b0fda94fbad75d7e P aa0b23d2e13e40008b2f8ff968a974f1: Deleting consensus metadata
I20260925 15:03:08.440871 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 23358
W20260925 15:03:08.459066 23282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:39079 (error 108)
I20260925 15:03:08.459792 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 23492
I20260925 15:03:08.461658 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.461203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50480)
0925 15:03:08.461371 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.461487 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 15:03:08.462886 23282 catalog_manager.cc:4584] TS aa0b23d2e13e40008b2f8ff968a974f1 (127.4.243.193:39079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f9f1128725d4067b0fda94fbad75d7e: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Sep 25 15:03:08 dist-test-slave-qxh0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:03:08.477926 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.460823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.4.243.194:41717)
0925 15:03:08.461015 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.461021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.461041 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:03:08.461105 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.461111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.461194 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.461314 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:08.462866 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.464061 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.465945 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.466619 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.466745 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.471852 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:08.471866 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:08.471882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:08.471923 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:08.475071 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.475786 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.475794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.475796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.475947 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.476335 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.476340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.476343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.476681 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:08.477035 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.477285 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.477507 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:03:08.478703 23615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:08.516594 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.516107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50482)
0925 15:03:08.516337 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.516432 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 15:03:08.517629 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.517279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50492)
0925 15:03:08.517426 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.517504 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:03:08.578328 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.577797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50508)
0925 15:03:08.578056 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.578150 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 15:03:08.579479 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.579138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50518)
0925 15:03:08.579297 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.579375 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:03:08.592810 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 23492
I20260925 15:03:08.611770 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 23625
Sep 25 15:03:08 dist-test-slave-qxh0 krb5kdc[23252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 15:03:08.626766 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.612564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.4.243.195:36853)
0925 15:03:08.612739 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:08.612744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:08.612837 (+ 93us) server_negotiation.cc:407] Connection header received
0925 15:03:08.612961 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:08.612965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:08.613035 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:08.613173 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:08.614430 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.615240 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.616526 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:08.616946 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:08.617042 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:08.621394 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:08.621410 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:08.621425 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:08.621458 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:08.624260 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.624830 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.624837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.624839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.624961 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:08.625251 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:08.625254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:08.625256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:08.625599 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:08.625865 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:08.626035 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:03:08.626197 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:03:08.627409 23748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:08.660329 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.659731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50530)
0925 15:03:08.659980 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.660099 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:03:08.661480 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.661010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50538)
0925 15:03:08.661265 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.661350 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 15:03:08.717471 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 23625
I20260925 15:03:08.734572 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 23265
I20260925 15:03:08.735746 23327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:08.848480 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 23265
I20260925 15:03:08.848599 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:08.848032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:39079 (local address 127.0.0.1:50550)
0925 15:03:08.848256 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:08.848382 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
2026-09-25T15:03:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4182 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:08Z Disabled control of system clock
I20260925 15:03:08.923434 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37553
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:37553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:03:09.245290 23806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:09.245597 23806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:09.245745 23806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:09.253115 23806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:09.253227 23806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:09.253294 23806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:09.253386 23806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:09.253453 23806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:09.262640 23806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:37553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:37553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:09.263973 23806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:09.265748 23806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:09.277201 23812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:09.277160 23814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:09.277258 23811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:09.278275 23806 server_base.cc:1034] running on GCE node
I20260925 15:03:09.279099 23806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:09.280435 23806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:09.281679 23806 hybrid_clock.cc:648] HybridClock initialized: now 1790348589281637 us; error 65 us; skew 500 ppm
I20260925 15:03:09.282053 23806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:09.284489 23806 webserver.cc:460] Webserver started at http://127.4.243.254:37627/ using document root <none> and password file <none>
I20260925 15:03:09.285190 23806 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:09.285329 23806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:09.285647 23806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:09.288527 23806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "32d3e2802704400a8196210bcf547a3b"
format_stamp: "Formatted at 2026-09-25 15:03:09 on dist-test-slave-qxh0"
server_key: "4f5837211c6f7791f4734499290597fd"
server_key_iv: "f4a24a9244af025ceb2411785ced1807"
server_key_version: "encryptionkey@0"
I20260925 15:03:09.289304 23806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "32d3e2802704400a8196210bcf547a3b"
format_stamp: "Formatted at 2026-09-25 15:03:09 on dist-test-slave-qxh0"
server_key: "4f5837211c6f7791f4734499290597fd"
server_key_iv: "f4a24a9244af025ceb2411785ced1807"
server_key_version: "encryptionkey@0"
I20260925 15:03:09.294204 23806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:03:09.297770 23820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:09.299135 23806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:09.299362 23806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "32d3e2802704400a8196210bcf547a3b"
format_stamp: "Formatted at 2026-09-25 15:03:09 on dist-test-slave-qxh0"
server_key: "4f5837211c6f7791f4734499290597fd"
server_key_iv: "f4a24a9244af025ceb2411785ced1807"
server_key_version: "encryptionkey@0"
I20260925 15:03:09.299585 23806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:09.317140 23806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:09.317991 23806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:09.318305 23806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:09.337844 23806 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:37553
I20260925 15:03:09.337864 23871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:37553 every 8 connection(s)
I20260925 15:03:09.339351 23806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:09.343067 23872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:09.346113 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 23806
I20260925 15:03:09.346305 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:09.346873 5071 external_mini_cluster.cc:1428] Setting key 65721d0b36455dbbde596eb3032fbdd7
I20260925 15:03:09.354820 23872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b: Bootstrap starting.
I20260925 15:03:09.357679 23872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:09.358750 23872 log.cc:824] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:09.360126 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:09.348764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.4.243.254:37553)
0925 15:03:09.349734 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:09.349744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:09.349783 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:03:09.351526 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:09.351552 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:09.352065 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:09.352591 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:09.353172 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:09.354400 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:09.355838 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:09.356510 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:09.356703 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:09.357861 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:09.357871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:09.357888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:09.358424 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:09.358512 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:09.358532 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:09.358684 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:09.358899 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:03:09.359211 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":361,"threads_started":1}
I20260925 15:03:09.361701 23872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b: No bootstrap required, opened a new log
I20260925 15:03:09.369120 23872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d3e2802704400a8196210bcf547a3b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37553 } }
I20260925 15:03:09.369434 23872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:09.369506 23872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d3e2802704400a8196210bcf547a3b, State: Initialized, Role: FOLLOWER
I20260925 15:03:09.369971 23872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d3e2802704400a8196210bcf547a3b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37553 } }
I20260925 15:03:09.370213 23872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:09.370399 23872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:09.370603 23872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:09.372665 23872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d3e2802704400a8196210bcf547a3b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37553 } }
I20260925 15:03:09.373128 23872 leader_election.cc:302] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32d3e2802704400a8196210bcf547a3b; no voters:
I20260925 15:03:09.373633 23872 leader_election.cc:288] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:09.373775 23877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:09.374873 23877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [term 1 LEADER]: Becoming Leader. State: Replica: 32d3e2802704400a8196210bcf547a3b, State: Running, Role: LEADER
I20260925 15:03:09.375384 23877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d3e2802704400a8196210bcf547a3b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37553 } }
I20260925 15:03:09.375777 23872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:09.378297 23878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32d3e2802704400a8196210bcf547a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d3e2802704400a8196210bcf547a3b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37553 } } }
I20260925 15:03:09.378274 23879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d3e2802704400a8196210bcf547a3b. Latest consensus state: current_term: 1 leader_uuid: "32d3e2802704400a8196210bcf547a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d3e2802704400a8196210bcf547a3b" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 37553 } } }
I20260925 15:03:09.378737 23879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:09.378734 23878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:09.380040 23885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:09.385159 23885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:09.393399 23885 catalog_manager.cc:1347] Generated new cluster ID: dac983d387464fdaa746885ce352a055
I20260925 15:03:09.393494 23885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:09.409399 23885 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:09.410763 23885 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:09.422572 23885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b: Generated new TSK 0
I20260925 15:03:09.423440 23885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:09.435158 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37553
--builtin_ntp_servers=127.4.243.212:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:03:09.783488 23896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:09.783738 23896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:09.783823 23896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:09.790827 23896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:09.790915 23896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:09.791015 23896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:03:09.799646 23896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:09.800853 23896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:09.802565 23896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:09.815802 23901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:09.815719 23904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:09.815685 23902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:09.816313 23896 server_base.cc:1034] running on GCE node
I20260925 15:03:09.817040 23896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:09.817914 23896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:09.819154 23896 hybrid_clock.cc:648] HybridClock initialized: now 1790348589819119 us; error 52 us; skew 500 ppm
I20260925 15:03:09.819463 23896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:09.821624 23896 webserver.cc:460] Webserver started at http://127.4.243.193:43679/ using document root <none> and password file <none>
I20260925 15:03:09.822288 23896 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:09.822409 23896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:09.822794 23896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:09.825446 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "d4e59ff266a34d52b122054f38b15064"
format_stamp: "Formatted at 2026-09-25 15:03:09 on dist-test-slave-qxh0"
server_key: "50c73f926ccd2c3cb90381c73fe2d27e"
server_key_iv: "68577c6cca8841f0b62d2fe657e1a02d"
server_key_version: "encryptionkey@0"
I20260925 15:03:09.826236 23896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "d4e59ff266a34d52b122054f38b15064"
format_stamp: "Formatted at 2026-09-25 15:03:09 on dist-test-slave-qxh0"
server_key: "50c73f926ccd2c3cb90381c73fe2d27e"
server_key_iv: "68577c6cca8841f0b62d2fe657e1a02d"
server_key_version: "encryptionkey@0"
I20260925 15:03:09.831156 23896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:03:09.834363 23910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:09.835722 23896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:03:09.835907 23896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "d4e59ff266a34d52b122054f38b15064"
format_stamp: "Formatted at 2026-09-25 15:03:09 on dist-test-slave-qxh0"
server_key: "50c73f926ccd2c3cb90381c73fe2d27e"
server_key_iv: "68577c6cca8841f0b62d2fe657e1a02d"
server_key_version: "encryptionkey@0"
I20260925 15:03:09.836127 23896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:09.850034 23896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:09.850961 23896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:09.851269 23896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:09.852219 23896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:09.853610 23896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:09.853726 23896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:09.853855 23896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:09.853947 23896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:09.886670 23896 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:42605
I20260925 15:03:09.886758 24022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:42605 every 8 connection(s)
I20260925 15:03:09.888008 23896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:03:09.893028 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 23896
I20260925 15:03:09.893191 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:03:09.893638 5071 external_mini_cluster.cc:1428] Setting key 7aed15b846e706169329abed15c8f854
I20260925 15:03:09.896667 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37553
--builtin_ntp_servers=127.4.243.212:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:03:09.902143 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:09.890506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:41331 (local address 127.4.243.254:37553)
0925 15:03:09.890992 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:09.890998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:09.892296 (+ 1298us) server_negotiation.cc:407] Connection header received
0925 15:03:09.894165 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:09.894170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:09.894243 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:09.894341 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:09.895733 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:09.896813 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:09.899360 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:09.899738 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:09.899845 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:09.901405 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:09.901409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:09.901413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:09.901430 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:09.901471 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:09.901482 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:09.901548 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:09.901801 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:03:09.901887 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
I20260925 15:03:09.903606 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:09.891253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37553 (local address 127.4.243.193:41331)
0925 15:03:09.892124 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:09.892158 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:09.893891 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:09.894578 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:09.894586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:09.894761 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:09.895552 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:09.895581 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:09.896996 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:09.897004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:09.899206 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:09.899235 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:09.899955 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:09.899963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:09.900284 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:09.901046 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:09.901051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:09.901104 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:09.901206 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:09.901577 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:09.901590 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:09.901600 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:03:09.901773 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 15:03:09.902179 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":266,"threads_started":1}
I20260925 15:03:09.905107 24023 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37553
I20260925 15:03:09.905406 24023 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:09.906139 24023 heartbeater.cc:511] Master 127.4.243.254:37553 requested a full tablet report, sending...
I20260925 15:03:09.908103 23837 ts_manager.cc:194] Registered new tserver with Master: d4e59ff266a34d52b122054f38b15064 (127.4.243.193:42605)
I20260925 15:03:09.909492 23837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.193:41331
W20260925 15:03:10.234409 24027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:10.234763 24027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:10.234903 24027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:10.242121 24027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:10.242242 24027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:10.242416 24027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:03:10.251870 24027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:10.253185 24027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:10.255015 24027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:10.268136 24032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:10.268100 24035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:10.268188 24033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:10.268702 24027 server_base.cc:1034] running on GCE node
I20260925 15:03:10.269349 24027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:10.270270 24027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:10.271494 24027 hybrid_clock.cc:648] HybridClock initialized: now 1790348590271437 us; error 70 us; skew 500 ppm
I20260925 15:03:10.271816 24027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:10.274065 24027 webserver.cc:460] Webserver started at http://127.4.243.194:38547/ using document root <none> and password file <none>
I20260925 15:03:10.274825 24027 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:10.274951 24027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:10.275270 24027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:10.278074 24027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "35e30360c7ff400f82aca03bd70912f7"
format_stamp: "Formatted at 2026-09-25 15:03:10 on dist-test-slave-qxh0"
server_key: "aaf520b82fc56534182779dec507912e"
server_key_iv: "29a7c2d4fa2220c3b7b153e0ea35e89b"
server_key_version: "encryptionkey@0"
I20260925 15:03:10.278889 24027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "35e30360c7ff400f82aca03bd70912f7"
format_stamp: "Formatted at 2026-09-25 15:03:10 on dist-test-slave-qxh0"
server_key: "aaf520b82fc56534182779dec507912e"
server_key_iv: "29a7c2d4fa2220c3b7b153e0ea35e89b"
server_key_version: "encryptionkey@0"
I20260925 15:03:10.284065 24027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:03:10.287667 24041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:10.288942 24027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:03:10.289112 24027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "35e30360c7ff400f82aca03bd70912f7"
format_stamp: "Formatted at 2026-09-25 15:03:10 on dist-test-slave-qxh0"
server_key: "aaf520b82fc56534182779dec507912e"
server_key_iv: "29a7c2d4fa2220c3b7b153e0ea35e89b"
server_key_version: "encryptionkey@0"
I20260925 15:03:10.289342 24027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:10.305924 24027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:10.306808 24027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:10.307099 24027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:10.308038 24027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:10.309450 24027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:10.309574 24027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:10.309706 24027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:10.309795 24027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:10.343144 24027 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:43403
I20260925 15:03:10.343185 24153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:43403 every 8 connection(s)
I20260925 15:03:10.344398 24027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:03:10.353590 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24027
I20260925 15:03:10.353783 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:03:10.354209 5071 external_mini_cluster.cc:1428] Setting key 80df0a9205ef4f1e320d53f4ef2dbb04
I20260925 15:03:10.356899 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.347118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:52479 (local address 127.4.243.254:37553)
0925 15:03:10.347292 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:10.347297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:10.348874 (+ 1577us) server_negotiation.cc:407] Connection header received
0925 15:03:10.350644 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:10.350651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:10.350739 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:10.350865 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:10.351892 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.352603 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.354157 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.354670 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.354803 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.356127 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:10.356133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:10.356137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:10.356162 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:10.356190 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:10.356201 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:10.356297 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:10.356508 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 15:03:10.356595 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:03:10.356988 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37553
--builtin_ntp_servers=127.4.243.212:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:03:10.357942 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.347774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37553 (local address 127.4.243.194:52479)
0925 15:03:10.348689 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:10.348750 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:10.350332 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:10.351085 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:10.351092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:10.351175 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:10.351709 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.351747 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.352758 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.352762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.354029 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.354041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.354891 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.354899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.355121 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.355826 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:10.355828 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:10.355865 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:10.355953 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:10.356296 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:10.356308 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:10.356318 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:03:10.356486 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 15:03:10.356877 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":271,"threads_started":1}
I20260925 15:03:10.359870 24154 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37553
I20260925 15:03:10.360162 24154 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:10.360863 24154 heartbeater.cc:511] Master 127.4.243.254:37553 requested a full tablet report, sending...
I20260925 15:03:10.362339 23837 ts_manager.cc:194] Registered new tserver with Master: 35e30360c7ff400f82aca03bd70912f7 (127.4.243.194:43403)
I20260925 15:03:10.363562 23837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.194:52479
W20260925 15:03:10.673537 24157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:10.673787 24157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:10.673873 24157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:10.680727 24157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:10.680820 24157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:10.680922 24157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:03:10.689992 24157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:37553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:10.691339 24157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:10.693084 24157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:10.705477 24162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:10.705495 24165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:10.705590 24163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:10.707289 24157 server_base.cc:1034] running on GCE node
I20260925 15:03:10.707851 24157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:10.708705 24157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:10.709928 24157 hybrid_clock.cc:648] HybridClock initialized: now 1790348590709899 us; error 65 us; skew 500 ppm
I20260925 15:03:10.710210 24157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:10.712375 24157 webserver.cc:460] Webserver started at http://127.4.243.195:39419/ using document root <none> and password file <none>
I20260925 15:03:10.713027 24157 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:10.713155 24157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:10.713469 24157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:10.716187 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "6af24f546c6c42199fb00c0e4bbe8b8e"
format_stamp: "Formatted at 2026-09-25 15:03:10 on dist-test-slave-qxh0"
server_key: "63fb987f0519065ea23767c8e048ade0"
server_key_iv: "55f65c8f4cb2987294322bf757e879be"
server_key_version: "encryptionkey@0"
I20260925 15:03:10.716948 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "6af24f546c6c42199fb00c0e4bbe8b8e"
format_stamp: "Formatted at 2026-09-25 15:03:10 on dist-test-slave-qxh0"
server_key: "63fb987f0519065ea23767c8e048ade0"
server_key_iv: "55f65c8f4cb2987294322bf757e879be"
server_key_version: "encryptionkey@0"
I20260925 15:03:10.721539 24157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:03:10.725044 24171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:10.726435 24157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:03:10.726672 24157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "6af24f546c6c42199fb00c0e4bbe8b8e"
format_stamp: "Formatted at 2026-09-25 15:03:10 on dist-test-slave-qxh0"
server_key: "63fb987f0519065ea23767c8e048ade0"
server_key_iv: "55f65c8f4cb2987294322bf757e879be"
server_key_version: "encryptionkey@0"
I20260925 15:03:10.726902 24157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:10.745028 24157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:10.745805 24157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:10.746105 24157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:10.747072 24157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:10.748504 24157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:10.748623 24157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:10.748751 24157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:10.748845 24157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:10.782374 24157 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:44165
I20260925 15:03:10.782437 24283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:44165 every 8 connection(s)
I20260925 15:03:10.783757 24157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:03:10.791306 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24157
I20260925 15:03:10.791465 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:03:10.791972 5071 external_mini_cluster.cc:1428] Setting key 49d1b2552f332c74881d4de2ca6287ca
I20260925 15:03:10.796571 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.786245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:57675 (local address 127.4.243.254:37553)
0925 15:03:10.786466 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:10.786473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:10.788093 (+ 1620us) server_negotiation.cc:407] Connection header received
0925 15:03:10.789792 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:10.789797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:10.789862 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:10.789948 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:10.790989 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.791841 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.793826 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.794311 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.794443 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.795754 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:10.795762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:10.795771 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:10.795793 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:10.795826 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:10.795837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:10.795930 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:10.796140 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:03:10.796234 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:03:10.797452 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.786880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:37553 (local address 127.4.243.195:57675)
0925 15:03:10.787940 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:10.787981 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:10.789560 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:10.790127 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:10.790134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:10.790213 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:10.790845 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.790863 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.791987 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.791995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.793623 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.793635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.794526 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.794532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.794780 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.795458 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:10.795461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:10.795496 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:10.795569 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:10.795929 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:10.795941 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:10.795951 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:03:10.796112 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 15:03:10.796470 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":343,"threads_started":1}
I20260925 15:03:10.798660 24284 heartbeater.cc:347] Connected to a master server at 127.4.243.254:37553
I20260925 15:03:10.798903 24284 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:10.799551 24284 heartbeater.cc:511] Master 127.4.243.254:37553 requested a full tablet report, sending...
I20260925 15:03:10.800773 23837 ts_manager.cc:194] Registered new tserver with Master: 6af24f546c6c42199fb00c0e4bbe8b8e (127.4.243.195:44165)
I20260925 15:03:10.801529 23837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.195:57675
I20260925 15:03:10.807005 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:03:10.816515 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.811052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.4.243.254:37553)
0925 15:03:10.811238 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:10.811247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:10.811664 (+ 417us) server_negotiation.cc:407] Connection header received
0925 15:03:10.811815 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:10.811819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:10.811884 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:10.811966 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:10.812409 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.813101 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.814364 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.814823 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.814923 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.815914 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:10.815919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:10.815922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:10.815939 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:10.815969 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:10.815980 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:10.816089 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:10.816234 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:03:10.816292 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:03:10.820312 23837 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:03:10.827077 23837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:03:10.866583 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.839467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43403 (local address 127.0.0.1:41764)
0925 15:03:10.855704 (+ 16237us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:10.855732 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:10.855917 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:10.859020 (+ 3103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:10.859031 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:10.859087 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:10.859623 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.859638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.861566 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.861574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.863690 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.863709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.865582 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.865592 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.865811 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.865866 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:03:10.865993 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:03:10.866180 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16074,"mutex_wait_us":1,"thread_start_us":15893,"threads_started":1}
I20260925 15:03:10.866662 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.839452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.4.243.194:43403)
0925 15:03:10.854963 (+ 15511us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:10.854978 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:10.856090 (+ 1112us) server_negotiation.cc:407] Connection header received
0925 15:03:10.856241 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:10.856254 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:10.857977 (+ 1723us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:10.858255 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:10.859870 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.861399 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.863856 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.865350 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.865485 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.865769 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:10.866079 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 15:03:10.866252 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15256,"thread_start_us":15535,"threads_started":1}
I20260925 15:03:10.868849 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.841408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38624)
0925 15:03:10.860200 (+ 18792us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:10.860220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:10.860333 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:10.861593 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:10.861596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:10.861615 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:10.862168 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.862180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.864118 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.864124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.866304 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.866318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.867971 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.867978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.868242 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.868279 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:03:10.868383 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:03:10.868509 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18667,"mutex_wait_us":97,"thread_start_us":18528,"threads_started":1}
I20260925 15:03:10.868885 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.839007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:44165 (local address 127.0.0.1:37620)
0925 15:03:10.839966 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:10.840102 (+ 136us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:10.840332 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:10.861259 (+ 20927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:10.861263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:10.861295 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:10.861708 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.861721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.864363 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.864370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.866142 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:10.866155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.867997 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.868005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:10.868266 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.868308 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:03:10.868401 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:03:10.868520 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":46,"thread_start_us":628,"threads_started":1}
I20260925 15:03:10.869123 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.841979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.4.243.193:42605)
0925 15:03:10.860795 (+ 18816us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:10.860805 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:10.860833 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:03:10.860934 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:10.860943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:10.861154 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:10.861342 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:10.862396 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.863977 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.866523 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.867758 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.867902 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.868124 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:10.868454 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:03:10.868602 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18647,"spinlock_wait_cycles":27520,"thread_start_us":18613,"threads_started":1}
I20260925 15:03:10.870172 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:10.839239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.4.243.195:44165)
0925 15:03:10.860694 (+ 21455us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:10.860704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:10.860743 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:03:10.860845 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:10.860855 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:10.861067 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:10.861269 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:10.861985 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.864211 (+ 2226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.866376 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:10.867786 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:10.868972 (+ 1186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:10.869586 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:10.869635 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:03:10.869771 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21264,"thread_start_us":21129,"threads_started":1}
I20260925 15:03:10.871203 24089 tablet_service.cc:1467] Processing CreateTablet for tablet ad2326ed651d4b658303c4806a6169b9 (DEFAULT_TABLE table=test-table [id=f0188f15bfe246cbbc6daa6e8e4b2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:10.872722 24089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2326ed651d4b658303c4806a6169b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:10.873202 24219 tablet_service.cc:1467] Processing CreateTablet for tablet ad2326ed651d4b658303c4806a6169b9 (DEFAULT_TABLE table=test-table [id=f0188f15bfe246cbbc6daa6e8e4b2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:10.873524 23958 tablet_service.cc:1467] Processing CreateTablet for tablet ad2326ed651d4b658303c4806a6169b9 (DEFAULT_TABLE table=test-table [id=f0188f15bfe246cbbc6daa6e8e4b2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:10.874325 24219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2326ed651d4b658303c4806a6169b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:10.874734 23958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2326ed651d4b658303c4806a6169b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:10.885782 24302 tablet_bootstrap.cc:492] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Bootstrap starting.
I20260925 15:03:10.889091 24303 tablet_bootstrap.cc:492] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Bootstrap starting.
I20260925 15:03:10.890121 24302 tablet_bootstrap.cc:654] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:10.891690 24302 log.cc:824] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:10.893543 24303 tablet_bootstrap.cc:654] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:10.894863 24302 tablet_bootstrap.cc:492] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: No bootstrap required, opened a new log
I20260925 15:03:10.894865 24303 log.cc:824] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:10.895164 24302 ts_tablet_manager.cc:1397] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 15:03:10.897682 24303 tablet_bootstrap.cc:492] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: No bootstrap required, opened a new log
I20260925 15:03:10.897981 24303 ts_tablet_manager.cc:1397] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 15:03:10.899263 24305 tablet_bootstrap.cc:492] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Bootstrap starting.
I20260925 15:03:10.902665 24305 tablet_bootstrap.cc:654] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:10.903867 24305 log.cc:824] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:10.905506 24302 raft_consensus.cc:348] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:10.905881 24302 raft_consensus.cc:374] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:10.905983 24302 raft_consensus.cc:729] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af24f546c6c42199fb00c0e4bbe8b8e, State: Initialized, Role: FOLLOWER
I20260925 15:03:10.906644 24302 consensus_queue.cc:260] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:10.906879 24305 tablet_bootstrap.cc:492] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: No bootstrap required, opened a new log
I20260925 15:03:10.907153 24305 ts_tablet_manager.cc:1397] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 15:03:10.907678 24284 heartbeater.cc:503] Master 127.4.243.254:37553 was elected leader, sending a full tablet report...
I20260925 15:03:10.908671 24303 raft_consensus.cc:348] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:10.909595 24302 ts_tablet_manager.cc:1428] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260925 15:03:10.909879 24303 raft_consensus.cc:374] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:10.910007 24303 raft_consensus.cc:729] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e59ff266a34d52b122054f38b15064, State: Initialized, Role: FOLLOWER
I20260925 15:03:10.910667 24303 consensus_queue.cc:260] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:10.911696 24023 heartbeater.cc:503] Master 127.4.243.254:37553 was elected leader, sending a full tablet report...
I20260925 15:03:10.914098 24303 ts_tablet_manager.cc:1428] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Time spent starting tablet: real 0.016s user 0.006s sys 0.006s
I20260925 15:03:10.916589 24305 raft_consensus.cc:348] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:10.916975 24305 raft_consensus.cc:374] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:10.917059 24305 raft_consensus.cc:729] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e30360c7ff400f82aca03bd70912f7, State: Initialized, Role: FOLLOWER
I20260925 15:03:10.917498 24305 consensus_queue.cc:260] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:10.918359 24154 heartbeater.cc:503] Master 127.4.243.254:37553 was elected leader, sending a full tablet report...
I20260925 15:03:10.918933 24305 ts_tablet_manager.cc:1428] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260925 15:03:11.035436 24285 tablet.cc:2369] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:03:11.096454 24155 tablet.cc:2369] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:03:11.140383 24024 tablet.cc:2369] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:03:11.171651 24308 raft_consensus.cc:482] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:03:11.172029 24308 raft_consensus.cc:504] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:11.173749 24308 leader_election.cc:288] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35e30360c7ff400f82aca03bd70912f7 (127.4.243.194:43403), d4e59ff266a34d52b122054f38b15064 (127.4.243.193:42605)
I20260925 15:03:11.179709 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.174224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:50935 (local address 127.4.243.194:43403)
0925 15:03:11.174419 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:11.174425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:11.174649 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:03:11.174920 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:11.174925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:11.174983 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:11.175079 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:11.175502 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.176268 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.178062 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.178873 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.179027 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.179099 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:11.179328 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 15:03:11.179448 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:03:11.179777 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.174212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43403 (local address 127.4.243.195:50935)
0925 15:03:11.174476 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.174496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:11.174742 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:11.175118 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:11.175123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:11.175205 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:11.175353 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:11.175361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.176394 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.176401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:11.177912 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:11.177923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.179017 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.179022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:11.179179 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.179202 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:03:11.179301 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:03:11.179428 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"spinlock_wait_cycles":37248}
I20260925 15:03:11.180204 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.174402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:45959 (local address 127.4.243.193:42605)
0925 15:03:11.174621 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:11.174627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:11.175107 (+ 480us) server_negotiation.cc:407] Connection header received
0925 15:03:11.175311 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:11.175321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:11.175386 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:11.175490 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:11.176024 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.176748 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.178497 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.179330 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.179485 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.179549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:11.179830 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:03:11.179954 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:03:11.180226 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.174246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.4.243.195:45959)
0925 15:03:11.175005 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.175019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:11.175173 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:11.175501 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:11.175506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:11.175527 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:11.175872 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:11.175881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.176898 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.176906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:11.178317 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:11.178329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.179501 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.179507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:11.179666 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.179696 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:03:11.179805 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:03:11.179916 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":463,"threads_started":1}
I20260925 15:03:11.180984 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2326ed651d4b658303c4806a6169b9" candidate_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e30360c7ff400f82aca03bd70912f7" is_pre_election: true
I20260925 15:03:11.181344 23978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2326ed651d4b658303c4806a6169b9" candidate_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e59ff266a34d52b122054f38b15064" is_pre_election: true
I20260925 15:03:11.181584 24109 raft_consensus.cc:2454] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af24f546c6c42199fb00c0e4bbe8b8e in term 0.
I20260925 15:03:11.181890 23978 raft_consensus.cc:2454] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af24f546c6c42199fb00c0e4bbe8b8e in term 0.
I20260925 15:03:11.182309 24175 leader_election.cc:302] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e30360c7ff400f82aca03bd70912f7, 6af24f546c6c42199fb00c0e4bbe8b8e; no voters:
I20260925 15:03:11.182741 24308 raft_consensus.cc:2781] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:03:11.182907 24308 raft_consensus.cc:482] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:03:11.183032 24308 raft_consensus.cc:3037] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:11.185137 24308 raft_consensus.cc:504] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:11.185746 24308 leader_election.cc:288] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [CANDIDATE]: Term 1 election: Requested vote from peers 35e30360c7ff400f82aca03bd70912f7 (127.4.243.194:43403), d4e59ff266a34d52b122054f38b15064 (127.4.243.193:42605)
I20260925 15:03:11.186151 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2326ed651d4b658303c4806a6169b9" candidate_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e30360c7ff400f82aca03bd70912f7"
I20260925 15:03:11.186208 23978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2326ed651d4b658303c4806a6169b9" candidate_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e59ff266a34d52b122054f38b15064"
I20260925 15:03:11.186367 24109 raft_consensus.cc:3037] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:11.186411 23978 raft_consensus.cc:3037] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:11.188580 24109 raft_consensus.cc:2454] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af24f546c6c42199fb00c0e4bbe8b8e in term 1.
I20260925 15:03:11.188585 23978 raft_consensus.cc:2454] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af24f546c6c42199fb00c0e4bbe8b8e in term 1.
I20260925 15:03:11.188970 24175 leader_election.cc:302] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e30360c7ff400f82aca03bd70912f7, 6af24f546c6c42199fb00c0e4bbe8b8e; no voters:
I20260925 15:03:11.189294 24308 raft_consensus.cc:2781] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:11.189705 24308 raft_consensus.cc:686] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 1 LEADER]: Becoming Leader. State: Replica: 6af24f546c6c42199fb00c0e4bbe8b8e, State: Running, Role: LEADER
I20260925 15:03:11.190282 24308 consensus_queue.cc:237] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } }
I20260925 15:03:11.194896 23836 catalog_manager.cc:5495] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6af24f546c6c42199fb00c0e4bbe8b8e (127.4.243.195). New cstate: current_term: 1 leader_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af24f546c6c42199fb00c0e4bbe8b8e" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 } health_report { overall_health: UNKNOWN } } }
I20260925 15:03:11.217195 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.212162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.4.243.195:44165)
0925 15:03:11.212340 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:11.212345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:11.212408 (+ 63us) server_negotiation.cc:407] Connection header received
0925 15:03:11.212632 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:11.212636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:11.212739 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:11.212852 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:11.213201 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.213936 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.215296 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.215681 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.215779 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.216259 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:11.216726 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:11.216851 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:11.216983 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:03:11.217017 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:03:11.224846 23978 raft_consensus.cc:1261] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 1 FOLLOWER]: Refusing update from remote peer 6af24f546c6c42199fb00c0e4bbe8b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:11.225020 24109 raft_consensus.cc:1261] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 1 FOLLOWER]: Refusing update from remote peer 6af24f546c6c42199fb00c0e4bbe8b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:11.225836 24312 consensus_queue.cc:1035] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e30360c7ff400f82aca03bd70912f7" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:11.226118 24308 consensus_queue.cc:1035] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e59ff266a34d52b122054f38b15064" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:11.237856 24315 mvcc.cc:205] Tried to move back new op lower bound from 7333267829647372288 to 7333267829519327232. Current Snapshot: MvccSnapshot[applied={T|T < 7333267829647372288}]
I20260925 15:03:11.239313 24317 mvcc.cc:205] Tried to move back new op lower bound from 7333267829647372288 to 7333267829519327232. Current Snapshot: MvccSnapshot[applied={T|T < 7333267829647372288}]
I20260925 15:03:11.250442 23837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:11.250772 23837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:11.254297 23837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32d3e2802704400a8196210bcf547a3b: Sending DeleteTablet for 3 replicas of tablet ad2326ed651d4b658303c4806a6169b9
I20260925 15:03:11.255519 23958 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2326ed651d4b658303c4806a6169b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:11 UTC) from {username='slave'} at 127.0.0.1:38624
I20260925 15:03:11.255607 24089 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2326ed651d4b658303c4806a6169b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:11 UTC) from {username='slave'} at 127.0.0.1:41764
I20260925 15:03:11.256083 24219 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2326ed651d4b658303c4806a6169b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:11 UTC) from {username='slave'} at 127.0.0.1:37620
I20260925 15:03:11.256716 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 23896
I20260925 15:03:11.268227 24327 tablet_replica.cc:330] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: stopping tablet replica
I20260925 15:03:11.269212 24327 raft_consensus.cc:2229] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:03:11.270370 24327 raft_consensus.cc:2258] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:11.273598 24327 ts_tablet_manager.cc:1905] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:11.273989 24325 tablet_replica.cc:330] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: stopping tablet replica
I20260925 15:03:11.274264 24324 tablet_replica.cc:330] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: stopping tablet replica
I20260925 15:03:11.274524 24325 raft_consensus.cc:2229] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:11.274732 24324 raft_consensus.cc:2229] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:11.275193 24324 raft_consensus.cc:2258] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:11.276736 24325 raft_consensus.cc:2258] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:11.278190 24324 ts_tablet_manager.cc:1905] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:11.280283 24325 ts_tablet_manager.cc:1905] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:11.282145 24327 ts_tablet_manager.cc:1918] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:11.282361 24327 log.cc:1196] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/ad2326ed651d4b658303c4806a6169b9
I20260925 15:03:11.282586 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.258459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.4.243.193:42605)
0925 15:03:11.258692 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:11.258702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:11.267726 (+ 9024us) server_negotiation.cc:407] Connection header received
0925 15:03:11.268243 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:11.268252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:11.268382 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:11.268473 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:11.269338 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.270924 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.275262 (+ 4338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.275956 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.276084 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.280826 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:11.280836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:11.280853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:11.281567 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:11.281630 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:11.281656 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:11.281759 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:11.282055 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 15:03:11.282175 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:03:11.283027 24327 ts_tablet_manager.cc:1939] T ad2326ed651d4b658303c4806a6169b9 P 6af24f546c6c42199fb00c0e4bbe8b8e: Deleting consensus metadata
I20260925 15:03:11.284039 24018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:11.284988 23821 catalog_manager.cc:4841] TS 6af24f546c6c42199fb00c0e4bbe8b8e (127.4.243.195:44165): tablet ad2326ed651d4b658303c4806a6169b9 (table test-table [id=f0188f15bfe246cbbc6daa6e8e4b2483]) successfully deleted
I20260925 15:03:11.292676 24325 ts_tablet_manager.cc:1918] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:11.292851 24325 log.cc:1196] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/ad2326ed651d4b658303c4806a6169b9
I20260925 15:03:11.293393 24325 ts_tablet_manager.cc:1939] T ad2326ed651d4b658303c4806a6169b9 P 35e30360c7ff400f82aca03bd70912f7: Deleting consensus metadata
I20260925 15:03:11.295248 23824 catalog_manager.cc:4841] TS 35e30360c7ff400f82aca03bd70912f7 (127.4.243.194:43403): tablet ad2326ed651d4b658303c4806a6169b9 (table test-table [id=f0188f15bfe246cbbc6daa6e8e4b2483]) successfully deleted
I20260925 15:03:11.387058 24324 ts_tablet_manager.cc:1918] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:11.387328 24324 log.cc:1196] T ad2326ed651d4b658303c4806a6169b9 P d4e59ff266a34d52b122054f38b15064: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/ad2326ed651d4b658303c4806a6169b9
I20260925 15:03:11.387691 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 23896
I20260925 15:03:11.406567 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24027
W20260925 15:03:11.406634 23822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:42605 (error 108)
I20260925 15:03:11.408942 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.408181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38638)
0925 15:03:11.408609 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.408761 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":225,"threads_started":1}
W20260925 15:03:11.409219 23822 catalog_manager.cc:4584] TS d4e59ff266a34d52b122054f38b15064 (127.4.243.193:42605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad2326ed651d4b658303c4806a6169b9: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
I20260925 15:03:11.414831 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.407839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.4.243.194:43403)
0925 15:03:11.408039 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:11.408046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:11.408066 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:03:11.408133 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:11.408139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:11.408248 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:11.408348 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:11.409017 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.410027 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.411916 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.412312 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.412412 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.413477 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:11.413486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:11.413499 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:11.414061 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:11.414106 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:11.414125 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:11.414230 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:11.414356 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 15:03:11.414441 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:03:11.415433 24149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:11.462973 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.462348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38646)
0925 15:03:11.462665 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.462782 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 15:03:11.464187 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.463756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38660)
0925 15:03:11.463923 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.464028 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:03:11.506371 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24027
I20260925 15:03:11.525008 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.524335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38666)
0925 15:03:11.524553 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.524642 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:03:11.526211 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.525827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38682)
0925 15:03:11.525985 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.526064 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:03:11.528824 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24157
I20260925 15:03:11.536139 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.529599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.4.243.195:44165)
0925 15:03:11.529849 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:11.529855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:11.529882 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:03:11.529980 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:11.529986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:11.530107 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:11.530237 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:11.530916 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.531748 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.533077 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:11.533509 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:11.533657 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:11.534780 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:11.534789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:11.534804 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:11.535394 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:11.535442 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:11.535467 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:11.535603 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:11.535721 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:03:11.535819 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:03:11.536983 24279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:11.607107 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.606643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38696)
0925 15:03:11.606867 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.606953 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:03:11.608161 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:11.607746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:42605 (local address 127.0.0.1:38706)
0925 15:03:11.607968 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:11.608037 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:42605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 15:03:11.655342 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24157
I20260925 15:03:11.674257 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 23806
I20260925 15:03:11.675372 23867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:11.784468 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 23806
2026-09-25T15:03:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:11 dist-test-slave-qxh0 krb5kdc[24336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:11 dist-test-slave-qxh0 krb5kdc[24336](info): set up 2 sockets
Sep 25 15:03:11 dist-test-slave-qxh0 krb5kdc[24336](info): commencing operation
krb5kdc: starting...
W20260925 15:03:12.886984 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:13 dist-test-slave-qxh0 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:13.121747 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:39699
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:33981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:39699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:13.440065 24349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:13.440354 24349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:13.440560 24349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:13.447868 24349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:13.447988 24349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:13.448096 24349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:13.448195 24349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:13.448282 24349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:13.458005 24349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:33981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:39699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:39699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:13.459393 24349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:13.460780 24349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:13.472115 24355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:13.472394 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:13.472534 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:13.473057 24349 server_base.cc:1034] running on GCE node
I20260925 15:03:13.473855 24349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:13.475317 24349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:13.476559 24349 hybrid_clock.cc:648] HybridClock initialized: now 1790348593476523 us; error 50 us; skew 500 ppm
I20260925 15:03:13.476900 24349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:13 dist-test-slave-qxh0 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:13.485452 24349 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:13.487238 24349 webserver.cc:460] Webserver started at http://127.4.243.254:45423/ using document root <none> and password file <none>
I20260925 15:03:13.487980 24349 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:13.488131 24349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:13.488445 24349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:13.491514 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "d85a83b3a3c84baa81bb7935a752bd3c"
format_stamp: "Formatted at 2026-09-25 15:03:13 on dist-test-slave-qxh0"
server_key: "4f837a48219c0d7b68bf1c5bb8848efd"
server_key_iv: "8d3ee1ccfcba69e471ab572431a99fef"
server_key_version: "encryptionkey@0"
I20260925 15:03:13.492278 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "d85a83b3a3c84baa81bb7935a752bd3c"
format_stamp: "Formatted at 2026-09-25 15:03:13 on dist-test-slave-qxh0"
server_key: "4f837a48219c0d7b68bf1c5bb8848efd"
server_key_iv: "8d3ee1ccfcba69e471ab572431a99fef"
server_key_version: "encryptionkey@0"
I20260925 15:03:13.497409 24349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:03:13.500735 24364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:13.501969 24349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:03:13.502151 24349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "d85a83b3a3c84baa81bb7935a752bd3c"
format_stamp: "Formatted at 2026-09-25 15:03:13 on dist-test-slave-qxh0"
server_key: "4f837a48219c0d7b68bf1c5bb8848efd"
server_key_iv: "8d3ee1ccfcba69e471ab572431a99fef"
server_key_version: "encryptionkey@0"
I20260925 15:03:13.502350 24349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:13.524061 24349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:13.528232 24349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:13.528515 24349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:13.546875 24349 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:39699
I20260925 15:03:13.546898 24415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:39699 every 8 connection(s)
I20260925 15:03:13.548194 24349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:13.551805 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:13.553490 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24349
I20260925 15:03:13.553670 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:13.554232 5071 external_mini_cluster.cc:1428] Setting key 65a950620bb627514295367192aea4d7
I20260925 15:03:13.566089 24416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c: Bootstrap starting.
Sep 25 15:03:13 dist-test-slave-qxh0 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:13.570338 24416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:13.571794 24416 log.cc:824] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:13.575313 24416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c: No bootstrap required, opened a new log
I20260925 15:03:13.578627 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:13.556111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.4.243.254:39699)
0925 15:03:13.556878 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:13.556889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:13.556932 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:03:13.558449 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:13.558468 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:13.558900 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:13.559203 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:13.561074 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:13.562396 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:13.564346 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:13.565057 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:13.565219 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:13.569997 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:13.570009 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:13.570028 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:13.570084 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:13.574075 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:13.574770 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:13.574780 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:13.574788 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:13.574980 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:13.575412 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:13.575417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:13.575420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:13.576107 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:13.576400 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:13.576783 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 15:03:13.577246 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":382,"threads_started":1}
I20260925 15:03:13.583755 24416 raft_consensus.cc:348] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39699 } }
I20260925 15:03:13.584070 24416 raft_consensus.cc:374] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:13.584137 24416 raft_consensus.cc:729] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85a83b3a3c84baa81bb7935a752bd3c, State: Initialized, Role: FOLLOWER
I20260925 15:03:13.584573 24416 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39699 } }
I20260925 15:03:13.584769 24416 raft_consensus.cc:388] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:13.584861 24416 raft_consensus.cc:482] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:13.584975 24416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:13.587100 24416 raft_consensus.cc:504] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39699 } }
I20260925 15:03:13.587553 24416 leader_election.cc:302] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85a83b3a3c84baa81bb7935a752bd3c; no voters:
I20260925 15:03:13.588168 24416 leader_election.cc:288] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:13.588315 24421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:13.589008 24421 raft_consensus.cc:686] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [term 1 LEADER]: Becoming Leader. State: Replica: d85a83b3a3c84baa81bb7935a752bd3c, State: Running, Role: LEADER
I20260925 15:03:13.589619 24421 consensus_queue.cc:237] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39699 } }
I20260925 15:03:13.590127 24416 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:13.592232 24422 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39699 } } }
I20260925 15:03:13.592547 24422 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:13.593443 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85a83b3a3c84baa81bb7935a752bd3c. Latest consensus state: current_term: 1 leader_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85a83b3a3c84baa81bb7935a752bd3c" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 39699 } } }
I20260925 15:03:13.594647 24427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:13.594807 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:13.599681 24427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:13.608852 24427 catalog_manager.cc:1347] Generated new cluster ID: 9f68097700cf493da278bde48a9ee5f0
I20260925 15:03:13.608981 24427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:13.630473 24427 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:13.631748 24427 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:13.646565 24427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c: Generated new TSK 0
I20260925 15:03:13.647722 24427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.243.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
Entry for principal kudu/127.4.243.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab.
I20260925 15:03:13.708889 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39699
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:33981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:14.032078 24442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:14.032341 24442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:14.032495 24442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:14.039522 24442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:14.039618 24442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:14.039817 24442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:03:14.049376 24442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:33981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.193.keytab
--principal=kudu/127.4.243.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:14.050717 24442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:14.052035 24442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:14.064881 24447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:14.064857 24448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:14.065040 24450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:14.065686 24442 server_base.cc:1034] running on GCE node
I20260925 15:03:14.066211 24442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:14.067130 24442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:14.068327 24442 hybrid_clock.cc:648] HybridClock initialized: now 1790348594068277 us; error 33 us; skew 500 ppm
I20260925 15:03:14.068578 24442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:14 dist-test-slave-qxh0 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348594, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:14.071733 24442 init.cc:377] Logged in from keytab as kudu/127.4.243.193@KRBTEST.COM (short username kudu)
I20260925 15:03:14.073720 24442 webserver.cc:460] Webserver started at http://127.4.243.193:46395/ using document root <none> and password file <none>
I20260925 15:03:14.074419 24442 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:14.074631 24442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:14.074928 24442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:14.077693 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "124a55547a6a435cb4899adee0377f1a"
format_stamp: "Formatted at 2026-09-25 15:03:14 on dist-test-slave-qxh0"
server_key: "ef944028e8b8c909c8a2c2a9bfc91edd"
server_key_iv: "62faf4c8f44b21870b9c65f53ef7249d"
server_key_version: "encryptionkey@0"
I20260925 15:03:14.078464 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "124a55547a6a435cb4899adee0377f1a"
format_stamp: "Formatted at 2026-09-25 15:03:14 on dist-test-slave-qxh0"
server_key: "ef944028e8b8c909c8a2c2a9bfc91edd"
server_key_iv: "62faf4c8f44b21870b9c65f53ef7249d"
server_key_version: "encryptionkey@0"
I20260925 15:03:14.083482 24442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:03:14.086903 24457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:14.088204 24442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:03:14.088392 24442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "124a55547a6a435cb4899adee0377f1a"
format_stamp: "Formatted at 2026-09-25 15:03:14 on dist-test-slave-qxh0"
server_key: "ef944028e8b8c909c8a2c2a9bfc91edd"
server_key_iv: "62faf4c8f44b21870b9c65f53ef7249d"
server_key_version: "encryptionkey@0"
I20260925 15:03:14.088591 24442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:14.116125 24442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:14.120026 24442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:14.120322 24442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:14.121289 24442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:14.122529 24442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:14.122654 24442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:14.122805 24442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:14.122880 24442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:14.155628 24442 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:35577
I20260925 15:03:14.155674 24569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:35577 every 8 connection(s)
I20260925 15:03:14.156898 24442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:03:14.161088 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24442
I20260925 15:03:14.161268 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:03:14.161686 5071 external_mini_cluster.cc:1428] Setting key c5be6a02c292e323e288e88395e334f7
Sep 25 15:03:14 dist-test-slave-qxh0 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348594, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.193@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:14.176882 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:14.159389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:57883 (local address 127.4.243.254:39699)
0925 15:03:14.159841 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:14.159846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:14.160559 (+ 713us) server_negotiation.cc:407] Connection header received
0925 15:03:14.162654 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:14.162660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:14.162758 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:14.162876 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:14.164567 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.165288 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.166943 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.167342 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.167486 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:14.171299 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:14.171304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:14.171307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:14.171341 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:14.173916 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:14.174658 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:14.174662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:14.174664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:14.174781 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:14.175216 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:14.175220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:14.175222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:14.175475 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:14.175631 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:14.176310 (+ 679us) server_negotiation.cc:299] Negotiation successful
0925 15:03:14.176529 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260925 15:03:14.177826 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:14.159761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39699 (local address 127.4.243.193:57883)
0925 15:03:14.160407 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:14.160442 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:14.162341 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:14.163113 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:14.163122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:14.163800 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:14.164389 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:14.164408 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.165441 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.165453 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:14.166791 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:14.166804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.167588 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.167596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:14.167840 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:14.168595 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:14.168617 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:14.171065 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:14.174054 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:14.174067 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:14.174083 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:14.174492 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:14.174891 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:14.174897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:14.174900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:14.175078 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:14.175604 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:14.175612 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:14.175937 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 15:03:14.176211 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 15:03:14.176732 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":247,"threads_started":1}
I20260925 15:03:14.179412 24570 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39699
I20260925 15:03:14.179754 24570 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:14.180578 24570 heartbeater.cc:511] Master 127.4.243.254:39699 requested a full tablet report, sending...
I20260925 15:03:14.183096 24381 ts_manager.cc:194] Registered new tserver with Master: 124a55547a6a435cb4899adee0377f1a (127.4.243.193:35577)
I20260925 15:03:14.185300 24381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.193@KRBTEST.COM'} at 127.4.243.193:57883
WARNING: no policy specified for kudu/127.4.243.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
Entry for principal kudu/127.4.243.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab.
I20260925 15:03:14.218919 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39699
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:33981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:14.538702 24576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:14.538965 24576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:14.539146 24576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:14.546124 24576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:14.546276 24576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:14.546504 24576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:03:14.556072 24576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:33981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.194.keytab
--principal=kudu/127.4.243.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:14.557256 24576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:14.558491 24576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:14.571345 24584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:14.571453 24582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:14.571448 24581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:14.572636 24576 server_base.cc:1034] running on GCE node
I20260925 15:03:14.573173 24576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:14.574018 24576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:14.575204 24576 hybrid_clock.cc:648] HybridClock initialized: now 1790348594575138 us; error 56 us; skew 500 ppm
I20260925 15:03:14.575520 24576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:14 dist-test-slave-qxh0 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348594, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:14.578734 24576 init.cc:377] Logged in from keytab as kudu/127.4.243.194@KRBTEST.COM (short username kudu)
I20260925 15:03:14.580169 24576 webserver.cc:460] Webserver started at http://127.4.243.194:32871/ using document root <none> and password file <none>
I20260925 15:03:14.580850 24576 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:14.580978 24576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:14.581284 24576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:14.584146 24576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "6e152e0fa30e465b87dcb495abbc66f2"
format_stamp: "Formatted at 2026-09-25 15:03:14 on dist-test-slave-qxh0"
server_key: "24ee6c0dc6b41504d55d6f37e4c1641e"
server_key_iv: "433c572c7c6e0ca808d399af83666831"
server_key_version: "encryptionkey@0"
I20260925 15:03:14.584911 24576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "6e152e0fa30e465b87dcb495abbc66f2"
format_stamp: "Formatted at 2026-09-25 15:03:14 on dist-test-slave-qxh0"
server_key: "24ee6c0dc6b41504d55d6f37e4c1641e"
server_key_iv: "433c572c7c6e0ca808d399af83666831"
server_key_version: "encryptionkey@0"
I20260925 15:03:14.589928 24576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:03:14.593379 24591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:14.594684 24576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:14.594856 24576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "6e152e0fa30e465b87dcb495abbc66f2"
format_stamp: "Formatted at 2026-09-25 15:03:14 on dist-test-slave-qxh0"
server_key: "24ee6c0dc6b41504d55d6f37e4c1641e"
server_key_iv: "433c572c7c6e0ca808d399af83666831"
server_key_version: "encryptionkey@0"
I20260925 15:03:14.595088 24576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:14.609898 24576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:14.613998 24576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:14.614223 24576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:14.615149 24576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:14.616322 24576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:14.616392 24576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:14.616475 24576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:14.616519 24576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:14.649396 24576 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:46083
I20260925 15:03:14.649430 24703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:46083 every 8 connection(s)
I20260925 15:03:14.650779 24576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:03:14.651381 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24576
I20260925 15:03:14.651548 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:03:14.652102 5071 external_mini_cluster.cc:1428] Setting key 0ec44627ec9e3f2eff77451dceeb4e34
Sep 25 15:03:14 dist-test-slave-qxh0 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348594, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.194@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.195@KRBTEST.COM; defaulting to no policy
I20260925 15:03:14.680092 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:14.654163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:52747 (local address 127.4.243.254:39699)
0925 15:03:14.654357 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:14.654363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:14.661122 (+ 6759us) server_negotiation.cc:407] Connection header received
0925 15:03:14.663381 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:14.663386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:14.663455 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:14.663547 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:14.665614 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.666321 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.668857 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.669448 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.669550 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:14.673582 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:14.673591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:14.673599 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:14.673645 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:14.677234 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:14.677915 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:14.677923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:14.677928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:14.678080 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:14.678531 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:14.678536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:14.678539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:14.678894 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:14.679007 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:14.679671 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 15:03:14.679840 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.195@KRBTEST.COM" created.
I20260925 15:03:14.681237 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:14.654745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39699 (local address 127.4.243.194:52747)
0925 15:03:14.660936 (+ 6191us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:14.660985 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:14.663081 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:14.663813 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:14.663827 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:14.664619 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:14.665438 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:14.665466 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.666472 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.666481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:14.668674 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:14.668692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:14.669679 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:14.669688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:14.670006 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:14.670739 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:14.670771 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:14.673358 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:14.677368 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:14.677374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:14.677384 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:14.677773 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:14.678191 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:14.678195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:14.678197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:14.678374 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:14.679009 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:14.679015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:14.679305 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 15:03:14.679596 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 15:03:14.679968 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5879,"thread_start_us":5683,"threads_started":1}
I20260925 15:03:14.682965 24704 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39699
I20260925 15:03:14.683265 24704 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:14.684018 24704 heartbeater.cc:511] Master 127.4.243.254:39699 requested a full tablet report, sending...
I20260925 15:03:14.685590 24381 ts_manager.cc:194] Registered new tserver with Master: 6e152e0fa30e465b87dcb495abbc66f2 (127.4.243.194:46083)
I20260925 15:03:14.686909 24381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.194@KRBTEST.COM'} at 127.4.243.194:52747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
Entry for principal kudu/127.4.243.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab.
I20260925 15:03:14.709717 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39699
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:33981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:15.029944 24709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:15.030210 24709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:15.030375 24709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:15.037591 24709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:15.037695 24709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:15.037873 24709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:03:15.047740 24709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:33981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.195.keytab
--principal=kudu/127.4.243.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:39699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:15.049085 24709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:15.050436 24709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:15.063126 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:15.063087 24715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:15.063162 24714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:15.063926 24709 server_base.cc:1034] running on GCE node
I20260925 15:03:15.064497 24709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:15.065407 24709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:15.066704 24709 hybrid_clock.cc:648] HybridClock initialized: now 1790348595066648 us; error 63 us; skew 500 ppm
I20260925 15:03:15.067035 24709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:15 dist-test-slave-qxh0 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348595, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:15.069967 24709 init.cc:377] Logged in from keytab as kudu/127.4.243.195@KRBTEST.COM (short username kudu)
I20260925 15:03:15.071478 24709 webserver.cc:460] Webserver started at http://127.4.243.195:45503/ using document root <none> and password file <none>
I20260925 15:03:15.072136 24709 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:15.072285 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:15.072584 24709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:15.075444 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "4602472e3ef944ef9733ad6aa175673b"
format_stamp: "Formatted at 2026-09-25 15:03:15 on dist-test-slave-qxh0"
server_key: "0add844a492f9ab4afec77a0050fc017"
server_key_iv: "cbb16420d60db3af8dcb5f2ae246f784"
server_key_version: "encryptionkey@0"
I20260925 15:03:15.076229 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "4602472e3ef944ef9733ad6aa175673b"
format_stamp: "Formatted at 2026-09-25 15:03:15 on dist-test-slave-qxh0"
server_key: "0add844a492f9ab4afec77a0050fc017"
server_key_iv: "cbb16420d60db3af8dcb5f2ae246f784"
server_key_version: "encryptionkey@0"
I20260925 15:03:15.081130 24709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:15.084403 24724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:15.085727 24709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:03:15.085915 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "4602472e3ef944ef9733ad6aa175673b"
format_stamp: "Formatted at 2026-09-25 15:03:15 on dist-test-slave-qxh0"
server_key: "0add844a492f9ab4afec77a0050fc017"
server_key_iv: "cbb16420d60db3af8dcb5f2ae246f784"
server_key_version: "encryptionkey@0"
I20260925 15:03:15.086129 24709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:15.104998 24709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:15.109269 24709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:15.109556 24709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:15.110468 24709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:15.111816 24709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:15.111917 24709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:15.112068 24709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:15.112151 24709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:15.145463 24709 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:34343
I20260925 15:03:15.145546 24836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:34343 every 8 connection(s)
I20260925 15:03:15.146829 24709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:03:15.151399 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24709
I20260925 15:03:15.151592 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:03:15.152132 5071 external_mini_cluster.cc:1428] Setting key 20f7ae606305b09e85c65d8a2f25ea3d
Sep 25 15:03:15 dist-test-slave-qxh0 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348595, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.195@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:15.167181 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.149419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:49681 (local address 127.4.243.254:39699)
0925 15:03:15.149641 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.149647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.150846 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 15:03:15.153248 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.153254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.153361 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.153486 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:15.155399 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.156157 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.157642 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.158028 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.158162 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.161495 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:15.161500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:15.161503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:15.161534 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:15.164000 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.164706 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.164715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.164718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.164843 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.165261 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.165267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.165275 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.165542 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:15.165718 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.166573 (+ 855us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.166826 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 15:03:15.168200 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.149925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:39699 (local address 127.4.243.195:49681)
0925 15:03:15.150687 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:15.150734 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:15.152972 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:15.153757 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:15.153768 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:15.154406 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:15.155154 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.155182 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.156307 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.156312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.157502 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.157513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.158254 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.158262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.158477 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.159223 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:15.159259 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:15.161304 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:15.164155 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:15.164163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:15.164177 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:15.164577 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:15.164962 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:15.164967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:15.164971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:15.165140 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:15.165676 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:15.165685 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:15.166168 (+ 483us) client_negotiation.cc:769] Sending connection context
0925 15:03:15.166472 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 15:03:15.167001 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":271,"threads_started":1}
I20260925 15:03:15.169516 24837 heartbeater.cc:347] Connected to a master server at 127.4.243.254:39699
I20260925 15:03:15.169765 24837 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:15.170406 24837 heartbeater.cc:511] Master 127.4.243.254:39699 requested a full tablet report, sending...
I20260925 15:03:15.171792 24381 ts_manager.cc:194] Registered new tserver with Master: 4602472e3ef944ef9733ad6aa175673b (127.4.243.195:34343)
I20260925 15:03:15.172657 24381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.243.195@KRBTEST.COM'} at 127.4.243.195:49681
I20260925 15:03:15.179776 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:03:15.188552 24570 heartbeater.cc:503] Master 127.4.243.254:39699 was elected leader, sending a full tablet report...
I20260925 15:03:15.196672 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.183884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.4.243.254:39699)
0925 15:03:15.184105 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.184113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.184456 (+ 343us) server_negotiation.cc:407] Connection header received
0925 15:03:15.184640 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.184646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.184710 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.184850 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:15.186264 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.187035 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.188339 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.188804 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.188940 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.191569 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:15.191582 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:15.191585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:15.191614 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:15.194045 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.194701 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.194706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.194710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.194848 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.195216 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.195222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.195225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.195537 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:15.195801 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.196146 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.196355 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:03:15.200294 24381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:03:15.207435 24381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:03:15.229753 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.219047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.4.243.193:35577)
0925 15:03:15.221921 (+ 2874us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.221929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.221955 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:03:15.222085 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.222093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.222321 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.223396 (+ 1075us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:15.223697 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.225371 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.227697 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.228593 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.228758 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.228987 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.229259 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.229406 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2710,"thread_start_us":2554,"threads_started":1}
I20260925 15:03:15.229903 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.218844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:35577 (local address 127.0.0.1:37330)
0925 15:03:15.220520 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:15.220562 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:15.220717 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:15.222554 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:15.222561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:15.222624 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:15.223181 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.223196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.225508 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.225513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.227543 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.227553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.228787 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.228796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.229046 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.229083 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:03:15.229210 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:03:15.229390 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1438,"mutex_wait_us":38,"spinlock_wait_cycles":7040,"thread_start_us":1632,"threads_started":1}
I20260925 15:03:15.230703 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.219724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:46083 (local address 127.0.0.1:49486)
0925 15:03:15.221253 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:15.221269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:15.221376 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:15.222468 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:15.222477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:15.222507 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:15.222941 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.222954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.224420 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.224426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.227120 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.227135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.229912 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.229919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.230092 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.230123 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:03:15.230211 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:03:15.230340 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409,"thread_start_us":1262,"threads_started":1}
I20260925 15:03:15.231392 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.220316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.4.243.194:46083)
0925 15:03:15.221640 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.221648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.221676 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:03:15.221771 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.221778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.221985 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.222197 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:15.223158 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.224293 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.227298 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.228758 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.228883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.229065 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.230928 (+ 1863us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.231086 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":1762,"threads_started":1}
I20260925 15:03:15.233839 24505 tablet_service.cc:1467] Processing CreateTablet for tablet 86843f858ec2405ba5c49c0e9fc3e127 (DEFAULT_TABLE table=test-table [id=b3f30879874e45ac9ebeff0f5c72d06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:15.235323 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.221262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:34343 (local address 127.0.0.1:41408)
0925 15:03:15.224874 (+ 3612us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:15.224891 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:15.225006 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:15.225563 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:15.225570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:15.225600 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:15.226090 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.226105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.227801 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.227806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.229620 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.229631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.232130 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.232136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.232298 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.234646 (+ 2348us) client_negotiation.cc:769] Sending connection context
0925 15:03:15.234768 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 15:03:15.234911 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3471,"mutex_wait_us":28,"thread_start_us":3312,"threads_started":1}
I20260925 15:03:15.234782 24639 tablet_service.cc:1467] Processing CreateTablet for tablet 86843f858ec2405ba5c49c0e9fc3e127 (DEFAULT_TABLE table=test-table [id=b3f30879874e45ac9ebeff0f5c72d06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:15.235508 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.221828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.4.243.195:34343)
0925 15:03:15.223503 (+ 1675us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.223513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.224990 (+ 1477us) server_negotiation.cc:407] Connection header received
0925 15:03:15.225163 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.225169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.225369 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.225528 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:15.226306 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.227624 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.229769 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.231923 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.232047 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.232273 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.234927 (+ 2654us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.235118 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1517,"thread_start_us":1373,"threads_started":1}
I20260925 15:03:15.236222 24505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86843f858ec2405ba5c49c0e9fc3e127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:15.236250 24639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86843f858ec2405ba5c49c0e9fc3e127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:15.238049 24772 tablet_service.cc:1467] Processing CreateTablet for tablet 86843f858ec2405ba5c49c0e9fc3e127 (DEFAULT_TABLE table=test-table [id=b3f30879874e45ac9ebeff0f5c72d06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:15.239131 24772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86843f858ec2405ba5c49c0e9fc3e127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:15.249920 24855 tablet_bootstrap.cc:492] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Bootstrap starting.
I20260925 15:03:15.251884 24857 tablet_bootstrap.cc:492] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Bootstrap starting.
I20260925 15:03:15.253621 24855 tablet_bootstrap.cc:654] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:15.254948 24855 log.cc:824] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:15.255306 24856 tablet_bootstrap.cc:492] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Bootstrap starting.
I20260925 15:03:15.255373 24857 tablet_bootstrap.cc:654] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:15.257828 24857 log.cc:824] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:15.258651 24856 tablet_bootstrap.cc:654] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:15.259604 24856 log.cc:824] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:15.260252 24855 tablet_bootstrap.cc:492] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: No bootstrap required, opened a new log
I20260925 15:03:15.260514 24855 ts_tablet_manager.cc:1397] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 15:03:15.261042 24857 tablet_bootstrap.cc:492] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: No bootstrap required, opened a new log
I20260925 15:03:15.261301 24857 ts_tablet_manager.cc:1397] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 15:03:15.262396 24856 tablet_bootstrap.cc:492] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: No bootstrap required, opened a new log
I20260925 15:03:15.262666 24856 ts_tablet_manager.cc:1397] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260925 15:03:15.267642 24855 raft_consensus.cc:348] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.268147 24855 raft_consensus.cc:374] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:15.268265 24855 raft_consensus.cc:729] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4602472e3ef944ef9733ad6aa175673b, State: Initialized, Role: FOLLOWER
I20260925 15:03:15.268846 24855 consensus_queue.cc:260] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.269855 24837 heartbeater.cc:503] Master 127.4.243.254:39699 was elected leader, sending a full tablet report...
I20260925 15:03:15.271514 24855 ts_tablet_manager.cc:1428] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 15:03:15.272014 24857 raft_consensus.cc:348] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.272423 24857 raft_consensus.cc:374] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:15.272541 24857 raft_consensus.cc:729] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124a55547a6a435cb4899adee0377f1a, State: Initialized, Role: FOLLOWER
I20260925 15:03:15.272816 24856 raft_consensus.cc:348] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.273233 24856 raft_consensus.cc:374] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:15.273330 24856 raft_consensus.cc:729] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e152e0fa30e465b87dcb495abbc66f2, State: Initialized, Role: FOLLOWER
I20260925 15:03:15.273144 24857 consensus_queue.cc:260] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.273952 24856 consensus_queue.cc:260] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.275758 24857 ts_tablet_manager.cc:1428] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 15:03:15.277031 24704 heartbeater.cc:503] Master 127.4.243.254:39699 was elected leader, sending a full tablet report...
I20260925 15:03:15.277536 24856 ts_tablet_manager.cc:1428] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260925 15:03:15.398489 24838 tablet.cc:2369] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:03:15.403466 24705 tablet.cc:2369] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:03:15.409175 24571 tablet.cc:2369] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:03:15.500900 24861 raft_consensus.cc:482] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:03:15.501187 24861 raft_consensus.cc:504] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.502815 24861 leader_election.cc:288] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124a55547a6a435cb4899adee0377f1a (127.4.243.193:35577), 6e152e0fa30e465b87dcb495abbc66f2 (127.4.243.194:46083)
I20260925 15:03:15.510015 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.503360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:60359 (local address 127.4.243.194:46083)
0925 15:03:15.503556 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.503564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.503689 (+ 125us) server_negotiation.cc:407] Connection header received
0925 15:03:15.503942 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.503948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.504027 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.504134 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:15.504529 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.505451 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.507925 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.509113 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.509279 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.509357 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.509636 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.509781 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:03:15.510406 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.503256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:42513 (local address 127.4.243.193:35577)
0925 15:03:15.503463 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.503470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.504143 (+ 673us) server_negotiation.cc:407] Connection header received
0925 15:03:15.504293 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.504298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.504371 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.504468 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:15.505011 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.505830 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.508163 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.509372 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.509557 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.509622 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.510051 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.510167 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:03:15.510483 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.503213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:46083 (local address 127.4.243.195:60359)
0925 15:03:15.503545 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:15.503575 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:15.503781 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:15.504169 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:15.504176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:15.504203 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:15.504396 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.504410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.505596 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.505601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.507771 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.507785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.509281 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.509289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.509470 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.509508 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:03:15.509612 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:03:15.510176 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":84}
I20260925 15:03:15.510552 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.503274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:35577 (local address 127.4.243.195:42513)
0925 15:03:15.504034 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:15.504050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:15.504162 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:15.504505 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:15.504509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:15.504531 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:15.504863 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.504874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.505976 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.505981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.508029 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:15.508039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.509785 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.509790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:15.509897 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.509918 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:03:15.510023 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:03:15.510144 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":29,"thread_start_us":429,"threads_started":1}
I20260925 15:03:15.511484 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86843f858ec2405ba5c49c0e9fc3e127" candidate_uuid: "4602472e3ef944ef9733ad6aa175673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e152e0fa30e465b87dcb495abbc66f2" is_pre_election: true
I20260925 15:03:15.511484 24525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86843f858ec2405ba5c49c0e9fc3e127" candidate_uuid: "4602472e3ef944ef9733ad6aa175673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124a55547a6a435cb4899adee0377f1a" is_pre_election: true
I20260925 15:03:15.512040 24525 raft_consensus.cc:2454] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4602472e3ef944ef9733ad6aa175673b in term 0.
I20260925 15:03:15.512038 24659 raft_consensus.cc:2454] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4602472e3ef944ef9733ad6aa175673b in term 0.
I20260925 15:03:15.512686 24728 leader_election.cc:302] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4602472e3ef944ef9733ad6aa175673b, 6e152e0fa30e465b87dcb495abbc66f2; no voters:
I20260925 15:03:15.513051 24861 raft_consensus.cc:2781] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:03:15.513195 24861 raft_consensus.cc:482] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:03:15.513320 24861 raft_consensus.cc:3037] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:15.515488 24861 raft_consensus.cc:504] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.516034 24861 leader_election.cc:288] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [CANDIDATE]: Term 1 election: Requested vote from peers 124a55547a6a435cb4899adee0377f1a (127.4.243.193:35577), 6e152e0fa30e465b87dcb495abbc66f2 (127.4.243.194:46083)
I20260925 15:03:15.516417 24525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86843f858ec2405ba5c49c0e9fc3e127" candidate_uuid: "4602472e3ef944ef9733ad6aa175673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124a55547a6a435cb4899adee0377f1a"
I20260925 15:03:15.516463 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86843f858ec2405ba5c49c0e9fc3e127" candidate_uuid: "4602472e3ef944ef9733ad6aa175673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e152e0fa30e465b87dcb495abbc66f2"
I20260925 15:03:15.516587 24525 raft_consensus.cc:3037] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:15.516680 24659 raft_consensus.cc:3037] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:15.518779 24659 raft_consensus.cc:2454] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4602472e3ef944ef9733ad6aa175673b in term 1.
I20260925 15:03:15.518833 24525 raft_consensus.cc:2454] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4602472e3ef944ef9733ad6aa175673b in term 1.
I20260925 15:03:15.519169 24728 leader_election.cc:302] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4602472e3ef944ef9733ad6aa175673b, 6e152e0fa30e465b87dcb495abbc66f2; no voters:
I20260925 15:03:15.519500 24861 raft_consensus.cc:2781] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:15.519999 24861 raft_consensus.cc:686] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 1 LEADER]: Becoming Leader. State: Replica: 4602472e3ef944ef9733ad6aa175673b, State: Running, Role: LEADER
I20260925 15:03:15.520447 24861 consensus_queue.cc:237] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } }
I20260925 15:03:15.524614 24380 catalog_manager.cc:5495] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4602472e3ef944ef9733ad6aa175673b (127.4.243.195). New cstate: current_term: 1 leader_uuid: "4602472e3ef944ef9733ad6aa175673b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4602472e3ef944ef9733ad6aa175673b" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 34343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20260925 15:03:15.594175 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.588539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.4.243.195:34343)
0925 15:03:15.588811 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.588815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.588833 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:03:15.588986 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.588990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.589075 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.589207 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:15.589509 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.590316 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.591836 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.592474 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.592570 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.593036 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:15.593588 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:15.593696 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.593823 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.593887 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 15:03:15.602069 24659 raft_consensus.cc:1261] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 1 FOLLOWER]: Refusing update from remote peer 4602472e3ef944ef9733ad6aa175673b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:15.602082 24525 raft_consensus.cc:1261] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 1 FOLLOWER]: Refusing update from remote peer 4602472e3ef944ef9733ad6aa175673b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:15.603003 24865 consensus_queue.cc:1035] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [LEADER]: Connected to new peer: Peer: permanent_uuid: "124a55547a6a435cb4899adee0377f1a" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:15.603331 24861 consensus_queue.cc:1035] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e152e0fa30e465b87dcb495abbc66f2" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:15.614055 24868 mvcc.cc:205] Tried to move back new op lower bound from 7333267847577358336 to 7333267847255158784. Current Snapshot: MvccSnapshot[applied={T|T < 7333267847577358336}]
I20260925 15:03:15.615391 24869 mvcc.cc:205] Tried to move back new op lower bound from 7333267847577358336 to 7333267847255158784. Current Snapshot: MvccSnapshot[applied={T|T < 7333267847577358336}]
I20260925 15:03:15.617731 24870 mvcc.cc:205] Tried to move back new op lower bound from 7333267847577358336 to 7333267847255158784. Current Snapshot: MvccSnapshot[applied={T|T < 7333267847577358336}]
I20260925 15:03:15.628933 24381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:15.629210 24381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:15.632740 24381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d85a83b3a3c84baa81bb7935a752bd3c: Sending DeleteTablet for 3 replicas of tablet 86843f858ec2405ba5c49c0e9fc3e127
I20260925 15:03:15.633708 24505 tablet_service.cc:1514] Processing DeleteTablet for tablet 86843f858ec2405ba5c49c0e9fc3e127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:15 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:37330
I20260925 15:03:15.633886 24639 tablet_service.cc:1514] Processing DeleteTablet for tablet 86843f858ec2405ba5c49c0e9fc3e127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:15 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:49486
I20260925 15:03:15.633891 24772 tablet_service.cc:1514] Processing DeleteTablet for tablet 86843f858ec2405ba5c49c0e9fc3e127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:15 UTC) from {username='kudu', principal='kudu/127.4.243.254@KRBTEST.COM'} at 127.0.0.1:41408
I20260925 15:03:15.637528 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24442
I20260925 15:03:15.645356 24879 tablet_replica.cc:330] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: stopping tablet replica
I20260925 15:03:15.645828 24879 raft_consensus.cc:2229] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:03:15.646186 24877 tablet_replica.cc:330] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: stopping tablet replica
I20260925 15:03:15.646463 24879 raft_consensus.cc:2258] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:15.646859 24877 raft_consensus.cc:2229] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:15.647414 24877 raft_consensus.cc:2258] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:15.647492 24878 tablet_replica.cc:330] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: stopping tablet replica
I20260925 15:03:15.647892 24878 raft_consensus.cc:2229] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:15.648382 24878 raft_consensus.cc:2258] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:15.649000 24879 ts_tablet_manager.cc:1905] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:15.649667 24877 ts_tablet_manager.cc:1905] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:15.657353 24877 ts_tablet_manager.cc:1918] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:15.657608 24877 log.cc:1196] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/86843f858ec2405ba5c49c0e9fc3e127
I20260925 15:03:15.658241 24877 ts_tablet_manager.cc:1939] T 86843f858ec2405ba5c49c0e9fc3e127 P 124a55547a6a435cb4899adee0377f1a: Deleting consensus metadata
I20260925 15:03:15.660001 24365 catalog_manager.cc:4841] TS 124a55547a6a435cb4899adee0377f1a (127.4.243.193:35577): tablet 86843f858ec2405ba5c49c0e9fc3e127 (table test-table [id=b3f30879874e45ac9ebeff0f5c72d06e]) successfully deleted
I20260925 15:03:15.660717 24878 ts_tablet_manager.cc:1905] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:15.661710 24879 ts_tablet_manager.cc:1918] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:15.661948 24879 log.cc:1196] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/86843f858ec2405ba5c49c0e9fc3e127
I20260925 15:03:15.662545 24879 ts_tablet_manager.cc:1939] T 86843f858ec2405ba5c49c0e9fc3e127 P 4602472e3ef944ef9733ad6aa175673b: Deleting consensus metadata
I20260925 15:03:15.664206 24365 catalog_manager.cc:4841] TS 4602472e3ef944ef9733ad6aa175673b (127.4.243.195:34343): tablet 86843f858ec2405ba5c49c0e9fc3e127 (table test-table [id=b3f30879874e45ac9ebeff0f5c72d06e]) successfully deleted
Sep 25 15:03:15 dist-test-slave-qxh0 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.193@KRBTEST.COM
I20260925 15:03:15.672375 24878 ts_tablet_manager.cc:1918] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:15.672632 24878 log.cc:1196] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/86843f858ec2405ba5c49c0e9fc3e127
I20260925 15:03:15.673197 24878 ts_tablet_manager.cc:1939] T 86843f858ec2405ba5c49c0e9fc3e127 P 6e152e0fa30e465b87dcb495abbc66f2: Deleting consensus metadata
I20260925 15:03:15.674975 24368 catalog_manager.cc:4841] TS 6e152e0fa30e465b87dcb495abbc66f2 (127.4.243.194:46083): tablet 86843f858ec2405ba5c49c0e9fc3e127 (table test-table [id=b3f30879874e45ac9ebeff0f5c72d06e]) successfully deleted
I20260925 15:03:15.677997 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.640776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.4.243.193:35577)
0925 15:03:15.641012 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.641022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.658424 (+ 17402us) server_negotiation.cc:407] Connection header received
0925 15:03:15.660504 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.660511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.660594 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.660678 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:15.662917 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.663938 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.665392 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.666029 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.666148 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.671404 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:15.671417 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:15.671435 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:15.671471 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:15.675222 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.675913 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.675920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.675922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.676037 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.676453 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.676463 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.676465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.676795 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:15.677101 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.677384 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.677598 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:03:15.678817 24565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:15.768108 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24442
I20260925 15:03:15.792196 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24576
Sep 25 15:03:15 dist-test-slave-qxh0 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.194@KRBTEST.COM
I20260925 15:03:15.809473 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.793225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.4.243.194:46083)
0925 15:03:15.793437 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.793444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.793467 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:03:15.793537 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.793543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.793631 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.793753 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:15.795166 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.796310 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.797568 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.798244 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.798365 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.802617 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:15.802635 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:15.802655 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:15.802735 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:15.806466 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.807302 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.807312 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.807316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.807483 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.807844 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.807849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.807853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.808367 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:15.808655 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.808868 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.809092 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:03:15.810256 24699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:15.903919 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24576
I20260925 15:03:15.924520 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24709
Sep 25 15:03:15 dist-test-slave-qxh0 krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.195@KRBTEST.COM
I20260925 15:03:15.943393 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:15.926002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.4.243.195:34343)
0925 15:03:15.926194 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:15.926199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:15.926224 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:03:15.926324 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:15.926377 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:15.926476 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:15.926628 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:15.928604 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.929901 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.931218 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:15.931951 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:15.932100 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:15.936383 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:15.936396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:15.936421 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:15.936459 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:15.940271 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.940987 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.941019 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.941023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.941200 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:15.941580 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:15.941586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:15.941590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:15.942052 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:15.942397 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:15.942727 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:03:15.942965 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:03:15.944232 24832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:16.063930 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24709
I20260925 15:03:16.083184 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24349
I20260925 15:03:16.084280 24411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:16.192413 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24349
2026-09-25T15:03:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4408 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:03:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:16Z Disabled control of system clock
I20260925 15:03:16.266283 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36279
--webserver_interface=127.4.243.254
--webserver_port=0
--builtin_ntp_servers=127.4.243.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:36279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:03:16.586297 24888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:16.586565 24888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:16.586707 24888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:16.593591 24888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:16.593678 24888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:16.593715 24888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:16.593776 24888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:16.593819 24888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:16.602528 24888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:36279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:36279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:16.603736 24888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:16.605471 24888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:16.616858 24893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:16.616827 24896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:16.616891 24894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:16.618036 24888 server_base.cc:1034] running on GCE node
I20260925 15:03:16.618918 24888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:16.620224 24888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:16.621449 24888 hybrid_clock.cc:648] HybridClock initialized: now 1790348596621402 us; error 57 us; skew 500 ppm
I20260925 15:03:16.621793 24888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:16.623929 24888 webserver.cc:460] Webserver started at http://127.4.243.254:36595/ using document root <none> and password file <none>
I20260925 15:03:16.624584 24888 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:16.624725 24888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:16.625034 24888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:16.627828 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "9304c9734dfe44cf9cb3621ec3a69383"
format_stamp: "Formatted at 2026-09-25 15:03:16 on dist-test-slave-qxh0"
server_key: "e254f9c1ece69f34138d0161fa787a96"
server_key_iv: "b1dcfdb0a3371b97228c070238517bf7"
server_key_version: "encryptionkey@0"
I20260925 15:03:16.628640 24888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "9304c9734dfe44cf9cb3621ec3a69383"
format_stamp: "Formatted at 2026-09-25 15:03:16 on dist-test-slave-qxh0"
server_key: "e254f9c1ece69f34138d0161fa787a96"
server_key_iv: "b1dcfdb0a3371b97228c070238517bf7"
server_key_version: "encryptionkey@0"
I20260925 15:03:16.633703 24888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:03:16.637059 24902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:16.638404 24888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:03:16.638651 24888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "9304c9734dfe44cf9cb3621ec3a69383"
format_stamp: "Formatted at 2026-09-25 15:03:16 on dist-test-slave-qxh0"
server_key: "e254f9c1ece69f34138d0161fa787a96"
server_key_iv: "b1dcfdb0a3371b97228c070238517bf7"
server_key_version: "encryptionkey@0"
I20260925 15:03:16.638864 24888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:16.659937 24888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:16.660808 24888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:16.661268 24888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:16.679831 24888 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:36279
I20260925 15:03:16.679872 24953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:36279 every 8 connection(s)
I20260925 15:03:16.681087 24888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:16.684751 24954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:16.686244 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24888
I20260925 15:03:16.686439 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:16.687021 5071 external_mini_cluster.cc:1428] Setting key c87ed3ebc6ccb51e39a72b4bd05250bc
I20260925 15:03:16.699725 24954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383: Bootstrap starting.
I20260925 15:03:16.701841 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:16.689017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.4.243.254:36279)
0925 15:03:16.690094 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:16.690122 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:16.690175 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:03:16.691904 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:16.691931 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:16.692524 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:16.692947 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:16.693492 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:16.695364 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:16.696673 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:16.697738 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:16.697903 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:16.699081 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:16.699095 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:16.699112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:16.699813 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:16.699910 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:16.699934 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:16.700047 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:16.700260 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 15:03:16.700572 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":343,"threads_started":1}
I20260925 15:03:16.703369 24954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:16.704437 24954 log.cc:824] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:16.707041 24954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383: No bootstrap required, opened a new log
I20260925 15:03:16.713984 24954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304c9734dfe44cf9cb3621ec3a69383" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36279 } }
I20260925 15:03:16.714337 24954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:16.714443 24954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9304c9734dfe44cf9cb3621ec3a69383, State: Initialized, Role: FOLLOWER
I20260925 15:03:16.715070 24954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304c9734dfe44cf9cb3621ec3a69383" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36279 } }
I20260925 15:03:16.715301 24954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:16.715443 24954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:16.715616 24954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:16.717669 24954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304c9734dfe44cf9cb3621ec3a69383" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36279 } }
I20260925 15:03:16.718154 24954 leader_election.cc:302] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9304c9734dfe44cf9cb3621ec3a69383; no voters:
I20260925 15:03:16.718765 24954 leader_election.cc:288] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:16.718896 24959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:16.720082 24959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [term 1 LEADER]: Becoming Leader. State: Replica: 9304c9734dfe44cf9cb3621ec3a69383, State: Running, Role: LEADER
I20260925 15:03:16.720582 24959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304c9734dfe44cf9cb3621ec3a69383" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36279 } }
I20260925 15:03:16.720824 24954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:16.723366 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9304c9734dfe44cf9cb3621ec3a69383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304c9734dfe44cf9cb3621ec3a69383" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36279 } } }
I20260925 15:03:16.723841 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:16.723343 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9304c9734dfe44cf9cb3621ec3a69383. Latest consensus state: current_term: 1 leader_uuid: "9304c9734dfe44cf9cb3621ec3a69383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9304c9734dfe44cf9cb3621ec3a69383" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 36279 } } }
I20260925 15:03:16.724119 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:16.725062 24967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:16.730405 24967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:16.738806 24967 catalog_manager.cc:1347] Generated new cluster ID: 80db60d384aa46eab5000a005669b4c1
I20260925 15:03:16.738929 24967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:16.748570 24967 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:16.749888 24967 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:16.755551 24967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383: Generated new TSK 0
I20260925 15:03:16.756253 24967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:16.764900 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.193:0
--local_ip_for_outbound_sockets=127.4.243.193
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36279
--builtin_ntp_servers=127.4.243.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:03:17.107259 24978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:17.107510 24978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:17.107595 24978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:17.114365 24978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:17.114449 24978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:17.114553 24978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.193
I20260925 15:03:17.123342 24978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.243.193
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.193
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:17.124434 24978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:17.126164 24978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:17.139005 24983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:17.139089 24986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:17.139117 24984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:17.139940 24978 server_base.cc:1034] running on GCE node
I20260925 15:03:17.140581 24978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:17.141547 24978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:17.142725 24978 hybrid_clock.cc:648] HybridClock initialized: now 1790348597142683 us; error 31 us; skew 500 ppm
I20260925 15:03:17.143081 24978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:17.145099 24978 webserver.cc:460] Webserver started at http://127.4.243.193:39673/ using document root <none> and password file <none>
I20260925 15:03:17.145766 24978 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:17.145876 24978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:17.146212 24978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:17.148952 24978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data/instance:
uuid: "eecb7762874a4f6681154a2af1aaf1ec"
format_stamp: "Formatted at 2026-09-25 15:03:17 on dist-test-slave-qxh0"
server_key: "90e88754d7cd45044c987275011fe562"
server_key_iv: "4e0ff236eff01cdd40295cad623fb485"
server_key_version: "encryptionkey@0"
I20260925 15:03:17.149739 24978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance:
uuid: "eecb7762874a4f6681154a2af1aaf1ec"
format_stamp: "Formatted at 2026-09-25 15:03:17 on dist-test-slave-qxh0"
server_key: "90e88754d7cd45044c987275011fe562"
server_key_iv: "4e0ff236eff01cdd40295cad623fb485"
server_key_version: "encryptionkey@0"
I20260925 15:03:17.154312 24978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:03:17.157464 24992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:17.158669 24978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:03:17.158849 24978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal
uuid: "eecb7762874a4f6681154a2af1aaf1ec"
format_stamp: "Formatted at 2026-09-25 15:03:17 on dist-test-slave-qxh0"
server_key: "90e88754d7cd45044c987275011fe562"
server_key_iv: "4e0ff236eff01cdd40295cad623fb485"
server_key_version: "encryptionkey@0"
I20260925 15:03:17.159054 24978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:17.183632 24978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:17.184484 24978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:17.184792 24978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:17.185828 24978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:17.187410 24978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:17.187494 24978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:17.187638 24978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:17.187737 24978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:17.220153 24978 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.193:44329
I20260925 15:03:17.220242 25104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.193:44329 every 8 connection(s)
I20260925 15:03:17.221710 24978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/data/info.pb
I20260925 15:03:17.231906 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 24978
I20260925 15:03:17.232092 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal/instance
I20260925 15:03:17.232511 5071 external_mini_cluster.cc:1428] Setting key bac2ad7efde76f2e66b2585f2b35cf48
I20260925 15:03:17.234191 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:17.224393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:44559 (local address 127.4.243.254:36279)
0925 15:03:17.224818 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:17.224823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:17.225969 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 15:03:17.227834 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:17.227840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:17.227930 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:17.228072 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:17.229091 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.229816 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.231411 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.231844 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.231985 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:17.233348 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:17.233353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:17.233356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:17.233372 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:17.233412 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:17.233431 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:17.233534 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:17.233753 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:03:17.233850 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260925 15:03:17.235148 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.194:0
--local_ip_for_outbound_sockets=127.4.243.194
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36279
--builtin_ntp_servers=127.4.243.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:03:17.235265 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:17.224950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36279 (local address 127.4.243.193:44559)
0925 15:03:17.225813 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:17.225858 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:17.227579 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:17.228288 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:17.228296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:17.228385 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:17.228931 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:17.228946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.229976 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.229981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:17.231257 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:17.231268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.232069 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.232076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:17.232344 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:17.233041 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:17.233044 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:17.233079 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:17.233154 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:17.233544 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:17.233557 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:17.233566 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:03:17.233741 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 15:03:17.234120 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":240,"threads_started":1}
I20260925 15:03:17.236699 25105 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36279
I20260925 15:03:17.237000 25105 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:17.237841 25105 heartbeater.cc:511] Master 127.4.243.254:36279 requested a full tablet report, sending...
I20260925 15:03:17.239850 24919 ts_manager.cc:194] Registered new tserver with Master: eecb7762874a4f6681154a2af1aaf1ec (127.4.243.193:44329)
I20260925 15:03:17.241263 24919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.193:44559
W20260925 15:03:17.554144 25109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:17.554392 25109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:17.554473 25109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:17.561363 25109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:17.561450 25109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:17.561609 25109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.194
I20260925 15:03:17.570416 25109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.243.194
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.194
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:17.571745 25109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:17.573506 25109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:17.586112 25117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:17.586175 25115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:17.586175 25114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:17.586999 25109 server_base.cc:1034] running on GCE node
I20260925 15:03:17.587558 25109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:17.588426 25109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:17.589660 25109 hybrid_clock.cc:648] HybridClock initialized: now 1790348597589621 us; error 53 us; skew 500 ppm
I20260925 15:03:17.590001 25109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:17.592199 25109 webserver.cc:460] Webserver started at http://127.4.243.194:33883/ using document root <none> and password file <none>
I20260925 15:03:17.592859 25109 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:17.592988 25109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:17.593300 25109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:17.596060 25109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data/instance:
uuid: "e190ef8dc1454b32a2341f919b763929"
format_stamp: "Formatted at 2026-09-25 15:03:17 on dist-test-slave-qxh0"
server_key: "bfbdad532fc4af58bc99a1a535d4bd1a"
server_key_iv: "9a1e95e4a0f23ea9cd66b85f9761a167"
server_key_version: "encryptionkey@0"
I20260925 15:03:17.596833 25109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance:
uuid: "e190ef8dc1454b32a2341f919b763929"
format_stamp: "Formatted at 2026-09-25 15:03:17 on dist-test-slave-qxh0"
server_key: "bfbdad532fc4af58bc99a1a535d4bd1a"
server_key_iv: "9a1e95e4a0f23ea9cd66b85f9761a167"
server_key_version: "encryptionkey@0"
I20260925 15:03:17.603785 25109 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260925 15:03:17.608130 25123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:17.609694 25109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:03:17.609889 25109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal
uuid: "e190ef8dc1454b32a2341f919b763929"
format_stamp: "Formatted at 2026-09-25 15:03:17 on dist-test-slave-qxh0"
server_key: "bfbdad532fc4af58bc99a1a535d4bd1a"
server_key_iv: "9a1e95e4a0f23ea9cd66b85f9761a167"
server_key_version: "encryptionkey@0"
I20260925 15:03:17.610175 25109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:17.627684 25109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:17.628439 25109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:17.628727 25109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:17.629647 25109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:17.631088 25109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:17.631194 25109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:17.631338 25109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:17.631428 25109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:17.664131 25109 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.194:43419
I20260925 15:03:17.664215 25235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.194:43419 every 8 connection(s)
I20260925 15:03:17.665391 25109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/data/info.pb
I20260925 15:03:17.670948 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25109
I20260925 15:03:17.671126 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal/instance
I20260925 15:03:17.671653 5071 external_mini_cluster.cc:1428] Setting key 9597877905ee857296b38b8f1ffe9730
I20260925 15:03:17.674154 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.243.195:0
--local_ip_for_outbound_sockets=127.4.243.195
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36279
--builtin_ntp_servers=127.4.243.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:03:17.677479 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:17.668074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.194:52409 (local address 127.4.243.254:36279)
0925 15:03:17.668256 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:17.668261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:17.669577 (+ 1316us) server_negotiation.cc:407] Connection header received
0925 15:03:17.671378 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:17.671384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:17.671466 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:17.671576 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:17.672705 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.673424 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.674990 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.675359 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.675485 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:17.676697 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:17.676701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:17.676704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:17.676719 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:17.676749 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:17.676759 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:17.676874 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:17.677088 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:03:17.677191 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:03:17.678403 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:17.668589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36279 (local address 127.4.243.194:52409)
0925 15:03:17.669440 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:17.669475 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:17.671045 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:17.671817 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:17.671825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:17.671907 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:17.672540 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:17.672562 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.673564 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.673569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:17.674842 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:17.674853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:17.675555 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:17.675561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:17.675765 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:17.676407 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:17.676409 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:17.676447 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:17.676513 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:17.676873 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:17.676884 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:17.676894 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:03:17.677077 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 15:03:17.677457 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":268,"threads_started":1}
I20260925 15:03:17.679834 25236 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36279
I20260925 15:03:17.680128 25236 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:17.680799 25236 heartbeater.cc:511] Master 127.4.243.254:36279 requested a full tablet report, sending...
I20260925 15:03:17.682260 24919 ts_manager.cc:194] Registered new tserver with Master: e190ef8dc1454b32a2341f919b763929 (127.4.243.194:43419)
I20260925 15:03:17.683347 24919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.194:52409
W20260925 15:03:17.994233 25239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:17.994542 25239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:17.994702 25239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:18.001734 25239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:18.001870 25239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:18.002054 25239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.243.195
I20260925 15:03:18.010987 25239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.243.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.243.195
--webserver_port=0
--tserver_master_addrs=127.4.243.254:36279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.243.195
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:18.012323 25239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:18.014258 25239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:18.026743 25245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:18.026774 25247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:18.026690 25244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:18.027446 25239 server_base.cc:1034] running on GCE node
I20260925 15:03:18.027968 25239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:18.028896 25239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:18.030144 25239 hybrid_clock.cc:648] HybridClock initialized: now 1790348598030091 us; error 39 us; skew 500 ppm
I20260925 15:03:18.030468 25239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:03:18.032617 25239 webserver.cc:460] Webserver started at http://127.4.243.195:46421/ using document root <none> and password file <none>
I20260925 15:03:18.033278 25239 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:18.033408 25239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:18.033735 25239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:18.036407 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data/instance:
uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3"
format_stamp: "Formatted at 2026-09-25 15:03:18 on dist-test-slave-qxh0"
server_key: "9d2c4b836b93135fcb5a9a5815d7985e"
server_key_iv: "1d3c8346493c325edc3727aeac80f0a3"
server_key_version: "encryptionkey@0"
I20260925 15:03:18.037176 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance:
uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3"
format_stamp: "Formatted at 2026-09-25 15:03:18 on dist-test-slave-qxh0"
server_key: "9d2c4b836b93135fcb5a9a5815d7985e"
server_key_iv: "1d3c8346493c325edc3727aeac80f0a3"
server_key_version: "encryptionkey@0"
I20260925 15:03:18.041956 25239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:03:18.045369 25253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:18.046686 25239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:03:18.046885 25239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal
uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3"
format_stamp: "Formatted at 2026-09-25 15:03:18 on dist-test-slave-qxh0"
server_key: "9d2c4b836b93135fcb5a9a5815d7985e"
server_key_iv: "1d3c8346493c325edc3727aeac80f0a3"
server_key_version: "encryptionkey@0"
I20260925 15:03:18.047118 25239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:18.059959 25239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:18.060668 25239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:18.060953 25239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:18.061848 25239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:03:18.063302 25239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:03:18.063421 25239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:18.063572 25239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:03:18.063655 25239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:18.096426 25239 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.195:40891
I20260925 15:03:18.096510 25365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.195:40891 every 8 connection(s)
I20260925 15:03:18.097697 25239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/data/info.pb
I20260925 15:03:18.099512 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25239
I20260925 15:03:18.099689 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal/instance
I20260925 15:03:18.100291 5071 external_mini_cluster.cc:1428] Setting key b70661a941b93975e170b0723ffdb274
I20260925 15:03:18.109853 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.100299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.195:43597 (local address 127.4.243.254:36279)
0925 15:03:18.100501 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.100506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.102379 (+ 1873us) server_negotiation.cc:407] Connection header received
0925 15:03:18.104150 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.104153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.104217 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.104299 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:18.105263 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.105992 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.107530 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.107930 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.108024 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.109192 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:18.109196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:18.109199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:18.109214 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:18.109244 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:18.109254 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:18.109328 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.109538 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.109614 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 15:03:18.110891 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.100928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:36279 (local address 127.4.243.195:43597)
0925 15:03:18.102219 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.102273 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:18.103909 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:18.104493 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:18.104501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:18.104583 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:18.105118 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.105133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.106118 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.106122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.107395 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.107406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.108101 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.108105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.108297 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.108936 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:03:18.108939 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:18.108973 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:03:18.109036 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:18.109335 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:18.109341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:18.109347 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:03:18.109511 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 15:03:18.109773 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":627,"threads_started":1}
I20260925 15:03:18.112099 25366 heartbeater.cc:347] Connected to a master server at 127.4.243.254:36279
I20260925 15:03:18.112392 25366 heartbeater.cc:464] Registering TS with master...
I20260925 15:03:18.113042 25366 heartbeater.cc:511] Master 127.4.243.254:36279 requested a full tablet report, sending...
I20260925 15:03:18.114696 24919 ts_manager.cc:194] Registered new tserver with Master: 8a47b2834c5c4a5e9de6d8cf4d7530c3 (127.4.243.195:40891)
I20260925 15:03:18.115447 5071 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:03:18.115629 24919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.243.195:43597
I20260925 15:03:18.125376 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.119996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.4.243.254:36279)
0925 15:03:18.120246 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.120253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.120581 (+ 328us) server_negotiation.cc:407] Connection header received
0925 15:03:18.120696 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.120700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.120810 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.120886 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:18.121331 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.122067 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.123373 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.123726 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.123813 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.124745 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:18.124749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:18.124752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:18.124766 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:18.124790 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:18.124801 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:18.124864 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.124980 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.125045 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:03:18.129174 24919 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:03:18.135845 24919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:03:18.168919 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.147657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.4.243.193:44329)
0925 15:03:18.148483 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.148492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.159476 (+ 10984us) server_negotiation.cc:407] Connection header received
0925 15:03:18.160711 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.160723 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.161043 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.161281 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:18.162485 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.164013 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.166573 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.167681 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.167850 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.168108 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.168390 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.168555 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":745,"threads_started":1}
I20260925 15:03:18.169826 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.147168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37662)
0925 15:03:18.159287 (+ 12119us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.159336 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:18.160398 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:18.161711 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:18.161722 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:18.161759 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:18.162298 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.162311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.164138 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.164145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.166373 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.166385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.167890 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.167896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.168113 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.168184 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 15:03:18.168346 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 15:03:18.168543 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11805,"mutex_wait_us":1,"thread_start_us":12350,"threads_started":1}
I20260925 15:03:18.170694 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.147314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:40891 (local address 127.0.0.1:47042)
0925 15:03:18.160270 (+ 12956us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.160293 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:18.160715 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:18.161921 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:18.161929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:18.161968 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:18.163295 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.163309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.165149 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.165157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.167630 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.167646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.169884 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.169892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.170095 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.170133 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:03:18.170202 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:03:18.170336 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12778,"mutex_wait_us":793,"thread_start_us":14886,"threads_started":1}
I20260925 15:03:18.171528 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.148166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.4.243.195:40891)
0925 15:03:18.148928 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.148938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.160415 (+ 11477us) server_negotiation.cc:407] Connection header received
0925 15:03:18.160938 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.160950 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.161272 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.161475 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:18.163531 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.165011 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.167868 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.169333 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.169442 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.169692 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.170984 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.171164 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":450,"threads_started":1}
I20260925 15:03:18.172814 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.149282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43419 (local address 127.0.0.1:59370)
0925 15:03:18.162523 (+ 13241us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.162568 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:18.162717 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:18.165270 (+ 2553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:18.165274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:18.165308 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:18.165875 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.165892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.167857 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.167862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.169948 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.169966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.172119 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.172125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.172294 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.172342 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:03:18.172441 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:03:18.172576 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13109,"thread_start_us":11138,"threads_started":1}
I20260925 15:03:18.173789 25040 tablet_service.cc:1467] Processing CreateTablet for tablet 865d000fb88440da8b0b4113799754c2 (DEFAULT_TABLE table=test-table [id=13b39482295c4b89af0a62ad9ff6624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:18.175024 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.149466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.4.243.194:43419)
0925 15:03:18.164632 (+ 15166us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.164641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.164668 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:03:18.164775 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.164802 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.165064 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.165270 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:18.166121 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.167700 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.170167 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.171611 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.171751 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.172006 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.174635 (+ 2629us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.174761 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14994,"thread_start_us":14869,"threads_started":1}
I20260925 15:03:18.175340 25040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865d000fb88440da8b0b4113799754c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:18.175117 25301 tablet_service.cc:1467] Processing CreateTablet for tablet 865d000fb88440da8b0b4113799754c2 (DEFAULT_TABLE table=test-table [id=13b39482295c4b89af0a62ad9ff6624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:18.176563 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865d000fb88440da8b0b4113799754c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:18.178028 25171 tablet_service.cc:1467] Processing CreateTablet for tablet 865d000fb88440da8b0b4113799754c2 (DEFAULT_TABLE table=test-table [id=13b39482295c4b89af0a62ad9ff6624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:03:18.179219 25171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865d000fb88440da8b0b4113799754c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:18.189533 25384 tablet_bootstrap.cc:492] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Bootstrap starting.
I20260925 15:03:18.192577 25385 tablet_bootstrap.cc:492] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Bootstrap starting.
I20260925 15:03:18.193290 25384 tablet_bootstrap.cc:654] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:18.194516 25384 log.cc:824] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:18.196039 25385 tablet_bootstrap.cc:654] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:18.196131 25386 tablet_bootstrap.cc:492] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Bootstrap starting.
I20260925 15:03:18.197192 25385 log.cc:824] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:18.199010 25384 tablet_bootstrap.cc:492] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: No bootstrap required, opened a new log
I20260925 15:03:18.199119 25386 tablet_bootstrap.cc:654] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:18.199311 25384 ts_tablet_manager.cc:1397] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 15:03:18.200145 25385 tablet_bootstrap.cc:492] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: No bootstrap required, opened a new log
I20260925 15:03:18.200256 25386 log.cc:824] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:18.200402 25385 ts_tablet_manager.cc:1397] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 15:03:18.203153 25386 tablet_bootstrap.cc:492] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: No bootstrap required, opened a new log
I20260925 15:03:18.203392 25386 ts_tablet_manager.cc:1397] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 15:03:18.206743 25384 raft_consensus.cc:348] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.207162 25384 raft_consensus.cc:374] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:18.207257 25384 raft_consensus.cc:729] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e190ef8dc1454b32a2341f919b763929, State: Initialized, Role: FOLLOWER
I20260925 15:03:18.207934 25384 consensus_queue.cc:260] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.211440 25236 heartbeater.cc:503] Master 127.4.243.254:36279 was elected leader, sending a full tablet report...
I20260925 15:03:18.211114 25385 raft_consensus.cc:348] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.211647 25385 raft_consensus.cc:374] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:18.211758 25385 raft_consensus.cc:729] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecb7762874a4f6681154a2af1aaf1ec, State: Initialized, Role: FOLLOWER
I20260925 15:03:18.212206 25384 ts_tablet_manager.cc:1428] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260925 15:03:18.212455 25385 consensus_queue.cc:260] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.213706 25105 heartbeater.cc:503] Master 127.4.243.254:36279 was elected leader, sending a full tablet report...
I20260925 15:03:18.214406 25385 ts_tablet_manager.cc:1428] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260925 15:03:18.215480 25386 raft_consensus.cc:348] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.216001 25386 raft_consensus.cc:374] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:18.216104 25386 raft_consensus.cc:729] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a47b2834c5c4a5e9de6d8cf4d7530c3, State: Initialized, Role: FOLLOWER
I20260925 15:03:18.216832 25386 consensus_queue.cc:260] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.218016 25366 heartbeater.cc:503] Master 127.4.243.254:36279 was elected leader, sending a full tablet report...
I20260925 15:03:18.218772 25386 ts_tablet_manager.cc:1428] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260925 15:03:18.224114 25106 tablet.cc:2369] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:03:18.349383 25367 tablet.cc:2369] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:03:18.391232 25391 raft_consensus.cc:482] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:03:18.391681 25391 raft_consensus.cc:504] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.393322 25391 leader_election.cc:288] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a47b2834c5c4a5e9de6d8cf4d7530c3 (127.4.243.195:40891), e190ef8dc1454b32a2341f919b763929 (127.4.243.194:43419)
I20260925 15:03:18.399854 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.393835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:39037 (local address 127.4.243.195:40891)
0925 15:03:18.394033 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.394041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.394474 (+ 433us) server_negotiation.cc:407] Connection header received
0925 15:03:18.394744 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.394751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.394833 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.394929 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:18.395405 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.396133 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.397903 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.398948 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.399115 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.399183 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.399513 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.399629 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:03:18.399960 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.393932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.243.193:57851 (local address 127.4.243.194:43419)
0925 15:03:18.394148 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.394154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.394945 (+ 791us) server_negotiation.cc:407] Connection header received
0925 15:03:18.395190 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.395195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.395263 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.395355 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:03:18.395848 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.396551 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.398243 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.399070 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.399245 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.399312 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.399600 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.399735 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:03:18.400012 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.393577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.195:40891 (local address 127.4.243.193:39037)
0925 15:03:18.394355 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.394375 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:18.394546 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:18.394949 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:18.394953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:18.394976 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:18.395269 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.395277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.396277 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.396287 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.397699 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.397711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.399130 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.399136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.399356 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.399380 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:03:18.399482 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:03:18.399604 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":40,"thread_start_us":442,"threads_started":1}
I20260925 15:03:18.400079 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.393581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.194:43419 (local address 127.4.243.193:57851)
0925 15:03:18.394846 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.394860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:18.395055 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:18.395372 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:18.395376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:18.395396 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:03:18.395710 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.395718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.396688 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.396693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.398074 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:18.398086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.399264 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.399271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:18.399454 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.399476 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:03:18.399574 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:03:18.399688 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":605,"spinlock_wait_cycles":22912,"thread_start_us":317,"threads_started":1}
I20260925 15:03:18.401031 25321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865d000fb88440da8b0b4113799754c2" candidate_uuid: "eecb7762874a4f6681154a2af1aaf1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" is_pre_election: true
I20260925 15:03:18.401031 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865d000fb88440da8b0b4113799754c2" candidate_uuid: "eecb7762874a4f6681154a2af1aaf1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e190ef8dc1454b32a2341f919b763929" is_pre_election: true
I20260925 15:03:18.401618 25191 raft_consensus.cc:2454] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eecb7762874a4f6681154a2af1aaf1ec in term 0.
I20260925 15:03:18.401618 25321 raft_consensus.cc:2454] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eecb7762874a4f6681154a2af1aaf1ec in term 0.
I20260925 15:03:18.402329 24996 leader_election.cc:302] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a47b2834c5c4a5e9de6d8cf4d7530c3, e190ef8dc1454b32a2341f919b763929, eecb7762874a4f6681154a2af1aaf1ec; no voters:
I20260925 15:03:18.402787 25391 raft_consensus.cc:2781] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:03:18.402963 25391 raft_consensus.cc:482] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:03:18.403069 25391 raft_consensus.cc:3037] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:18.405182 25391 raft_consensus.cc:504] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.405815 25391 leader_election.cc:288] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [CANDIDATE]: Term 1 election: Requested vote from peers 8a47b2834c5c4a5e9de6d8cf4d7530c3 (127.4.243.195:40891), e190ef8dc1454b32a2341f919b763929 (127.4.243.194:43419)
I20260925 15:03:18.406363 25321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865d000fb88440da8b0b4113799754c2" candidate_uuid: "eecb7762874a4f6681154a2af1aaf1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3"
I20260925 15:03:18.406373 25191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865d000fb88440da8b0b4113799754c2" candidate_uuid: "eecb7762874a4f6681154a2af1aaf1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e190ef8dc1454b32a2341f919b763929"
I20260925 15:03:18.406576 25321 raft_consensus.cc:3037] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:18.406625 25191 raft_consensus.cc:3037] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:18.408790 25321 raft_consensus.cc:2454] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eecb7762874a4f6681154a2af1aaf1ec in term 1.
I20260925 15:03:18.408793 25191 raft_consensus.cc:2454] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eecb7762874a4f6681154a2af1aaf1ec in term 1.
I20260925 15:03:18.409242 24996 leader_election.cc:302] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a47b2834c5c4a5e9de6d8cf4d7530c3, eecb7762874a4f6681154a2af1aaf1ec; no voters:
I20260925 15:03:18.409555 25391 raft_consensus.cc:2781] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:18.410022 25391 raft_consensus.cc:686] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 1 LEADER]: Becoming Leader. State: Replica: eecb7762874a4f6681154a2af1aaf1ec, State: Running, Role: LEADER
I20260925 15:03:18.410506 25391 consensus_queue.cc:237] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } }
I20260925 15:03:18.415374 24918 catalog_manager.cc:5495] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec reported cstate change: term changed from 0 to 1, leader changed from <none> to eecb7762874a4f6681154a2af1aaf1ec (127.4.243.193). New cstate: current_term: 1 leader_uuid: "eecb7762874a4f6681154a2af1aaf1ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eecb7762874a4f6681154a2af1aaf1ec" member_type: VOTER last_known_addr { host: "127.4.243.193" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 } health_report { overall_health: UNKNOWN } } }
W20260925 15:03:18.417428 25237 tablet.cc:2369] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:03:18.450822 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.445391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.4.243.193:44329)
0925 15:03:18.445583 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.445587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.445734 (+ 147us) server_negotiation.cc:407] Connection header received
0925 15:03:18.445955 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.445959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.446063 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.446186 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:18.446490 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.447196 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.448679 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.449202 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.449302 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.449720 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:18.450249 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:18.450390 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.450464 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.450551 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:03:18.459089 25321 raft_consensus.cc:1261] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 1 FOLLOWER]: Refusing update from remote peer eecb7762874a4f6681154a2af1aaf1ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:18.459089 25191 raft_consensus.cc:1261] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 1 FOLLOWER]: Refusing update from remote peer eecb7762874a4f6681154a2af1aaf1ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:18.460045 25395 consensus_queue.cc:1035] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e190ef8dc1454b32a2341f919b763929" member_type: VOTER last_known_addr { host: "127.4.243.194" port: 43419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:18.460388 25391 consensus_queue.cc:1035] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a47b2834c5c4a5e9de6d8cf4d7530c3" member_type: VOTER last_known_addr { host: "127.4.243.195" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:18.472311 25398 mvcc.cc:205] Tried to move back new op lower bound from 7333267859277754368 to 7333267859093630976. Current Snapshot: MvccSnapshot[applied={T|T < 7333267859277754368}]
I20260925 15:03:18.472378 25400 mvcc.cc:205] Tried to move back new op lower bound from 7333267859277754368 to 7333267859093630976. Current Snapshot: MvccSnapshot[applied={T|T < 7333267859277754368}]
I20260925 15:03:18.476598 25399 mvcc.cc:205] Tried to move back new op lower bound from 7333267859277754368 to 7333267859093630976. Current Snapshot: MvccSnapshot[applied={T|T < 7333267859277754368}]
I20260925 15:03:18.485883 24918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:18.486176 24918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:03:18.490025 24918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9304c9734dfe44cf9cb3621ec3a69383: Sending DeleteTablet for 3 replicas of tablet 865d000fb88440da8b0b4113799754c2
I20260925 15:03:18.491302 25301 tablet_service.cc:1514] Processing DeleteTablet for tablet 865d000fb88440da8b0b4113799754c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:18 UTC) from {username='slave'} at 127.0.0.1:47042
I20260925 15:03:18.491305 25040 tablet_service.cc:1514] Processing DeleteTablet for tablet 865d000fb88440da8b0b4113799754c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:18 UTC) from {username='slave'} at 127.0.0.1:37662
I20260925 15:03:18.491374 25171 tablet_service.cc:1514] Processing DeleteTablet for tablet 865d000fb88440da8b0b4113799754c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:03:18 UTC) from {username='slave'} at 127.0.0.1:59370
I20260925 15:03:18.495774 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24978
I20260925 15:03:18.501389 25410 tablet_replica.cc:330] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: stopping tablet replica
I20260925 15:03:18.501855 25410 raft_consensus.cc:2229] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:18.502372 25410 raft_consensus.cc:2258] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:18.504705 25410 ts_tablet_manager.cc:1905] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:18.511109 25408 tablet_replica.cc:330] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: stopping tablet replica
I20260925 15:03:18.511546 25408 raft_consensus.cc:2229] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:03:18.512095 25408 raft_consensus.cc:2258] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:18.514876 25409 tablet_replica.cc:330] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: stopping tablet replica
I20260925 15:03:18.515259 25409 raft_consensus.cc:2229] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:03:18.515887 25410 ts_tablet_manager.cc:1918] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:18.515902 25409 raft_consensus.cc:2258] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:03:18.516306 25408 ts_tablet_manager.cc:1905] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:18.516136 25410 log.cc:1196] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-1/wal/wals/865d000fb88440da8b0b4113799754c2
I20260925 15:03:18.516999 25410 ts_tablet_manager.cc:1939] T 865d000fb88440da8b0b4113799754c2 P e190ef8dc1454b32a2341f919b763929: Deleting consensus metadata
I20260925 15:03:18.518972 24903 catalog_manager.cc:4841] TS e190ef8dc1454b32a2341f919b763929 (127.4.243.194:43419): tablet 865d000fb88440da8b0b4113799754c2 (table test-table [id=13b39482295c4b89af0a62ad9ff6624c]) successfully deleted
I20260925 15:03:18.520130 25409 ts_tablet_manager.cc:1905] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:03:18.523739 25408 ts_tablet_manager.cc:1918] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:18.523887 25408 log.cc:1196] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-2/wal/wals/865d000fb88440da8b0b4113799754c2
I20260925 15:03:18.524377 25408 ts_tablet_manager.cc:1939] T 865d000fb88440da8b0b4113799754c2 P 8a47b2834c5c4a5e9de6d8cf4d7530c3: Deleting consensus metadata
I20260925 15:03:18.524945 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.497218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.4.243.193:44329)
0925 15:03:18.497442 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.497451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.511480 (+ 14029us) server_negotiation.cc:407] Connection header received
0925 15:03:18.511892 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.511905 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.512037 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.512129 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:18.513029 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.514657 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.516411 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.517095 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.517205 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.523053 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:18.523064 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:18.523079 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:18.523879 (+ 800us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:18.523953 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:18.523976 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:18.524068 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.524423 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.524541 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:03:18.526144 24906 catalog_manager.cc:4841] TS 8a47b2834c5c4a5e9de6d8cf4d7530c3 (127.4.243.195:40891): tablet 865d000fb88440da8b0b4113799754c2 (table test-table [id=13b39482295c4b89af0a62ad9ff6624c]) successfully deleted
I20260925 15:03:18.526328 25100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:18.622156 25409 ts_tablet_manager.cc:1918] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:03:18.622452 25409 log.cc:1196] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Deleting WAL directory at /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284481048-5071-0/minicluster-data/ts-0/wal/wals/865d000fb88440da8b0b4113799754c2
I20260925 15:03:18.623222 25409 ts_tablet_manager.cc:1939] T 865d000fb88440da8b0b4113799754c2 P eecb7762874a4f6681154a2af1aaf1ec: Deleting consensus metadata
I20260925 15:03:18.623310 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24978
I20260925 15:03:18.641295 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25109
W20260925 15:03:18.641283 24904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.243.193:44329 (error 108)
I20260925 15:03:18.643088 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.642545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37702)
0925 15:03:18.642758 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.642919 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 15:03:18.643352 24904 catalog_manager.cc:4584] TS eecb7762874a4f6681154a2af1aaf1ec (127.4.243.193:44329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 865d000fb88440da8b0b4113799754c2: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
I20260925 15:03:18.648043 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.642395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.4.243.194:43419)
0925 15:03:18.642605 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.642614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.642634 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:03:18.642701 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.642706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.642845 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.642961 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:18.643347 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.644141 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.645340 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.645739 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.645843 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.646856 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:18.646877 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:18.646891 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:18.647445 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:18.647493 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:18.647513 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:18.647615 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.647694 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.647790 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:03:18.648777 25231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:18.696815 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.696358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37712)
0925 15:03:18.696560 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.696652 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 15:03:18.697837 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.697527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37714)
0925 15:03:18.697655 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.697721 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:03:18.736536 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25109
I20260925 15:03:18.753057 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25239
I20260925 15:03:18.758415 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.757907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37722)
0925 15:03:18.758143 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.758234 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:03:18.759482 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.759099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37730)
0925 15:03:18.759276 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.759350 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:03:18.759682 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.753769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.4.243.195:40891)
0925 15:03:18.753929 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:18.753933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:18.753978 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:03:18.754118 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:18.754122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:18.754203 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:18.754297 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:18.754700 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.755402 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.756607 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:18.756970 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:18.757051 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:18.758166 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:18.758178 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:18.758198 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:18.758930 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:18.758975 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:18.758995 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:18.759084 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:18.759255 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:03:18.759359 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:03:18.760252 25361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:18.839975 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.839392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37742)
0925 15:03:18.839628 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.839775 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:03:18.841152 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:18.840688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.193:44329 (local address 127.0.0.1:37744)
0925 15:03:18.840939 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:18.841028 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.193:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260925 15:03:18.848366 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25239
I20260925 15:03:18.864923 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 24888
I20260925 15:03:18.865974 24949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:18.972715 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 24888
2026-09-25T15:03:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2782 ms)
[----------] 6 tests from EncryptionPolicyTest (21300 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:19 dist-test-slave-qxh0 krb5kdc[25419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:19 dist-test-slave-qxh0 krb5kdc[25419](info): set up 2 sockets
Sep 25 15:03:19 dist-test-slave-qxh0 krb5kdc[25419](info): commencing operation
krb5kdc: starting...
W20260925 15:03:20.067194 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:20 dist-test-slave-qxh0 krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:20.326543 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:20.646011 25432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:20.646292 25432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:20.646507 25432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:20.653790 25432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:20.653934 25432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:20.654028 25432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:20.654120 25432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:20.654230 25432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:20.663784 25432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:20.665064 25432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:20.666395 25432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:20.677591 25440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:20.677587 25438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:20.677834 25437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:20.678489 25432 server_base.cc:1034] running on GCE node
I20260925 15:03:20.679509 25432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:20.680794 25432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:20.681994 25432 hybrid_clock.cc:648] HybridClock initialized: now 1790348600681946 us; error 49 us; skew 500 ppm
I20260925 15:03:20.682325 25432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:20 dist-test-slave-qxh0 krb5kdc[25419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:20.685410 25432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:20.686815 25432 webserver.cc:460] Webserver started at http://127.0.0.1:42391/ using document root <none> and password file <none>
I20260925 15:03:20.687456 25432 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:20.687561 25432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:20.687810 25432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:20.690809 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "9897f28d44894ed68ac2cf650168c9bd"
format_stamp: "Formatted at 2026-09-25 15:03:20 on dist-test-slave-qxh0"
server_key: "af60be3a9adfd20f013e8b13cd943561"
server_key_iv: "a77c94833575757901bc0544b20f3f4b"
server_key_version: "encryptionkey@0"
I20260925 15:03:20.691524 25432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "9897f28d44894ed68ac2cf650168c9bd"
format_stamp: "Formatted at 2026-09-25 15:03:20 on dist-test-slave-qxh0"
server_key: "af60be3a9adfd20f013e8b13cd943561"
server_key_iv: "a77c94833575757901bc0544b20f3f4b"
server_key_version: "encryptionkey@0"
I20260925 15:03:20.696357 25432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:20.699635 25447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:20.700891 25432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:20.701076 25432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "9897f28d44894ed68ac2cf650168c9bd"
format_stamp: "Formatted at 2026-09-25 15:03:20 on dist-test-slave-qxh0"
server_key: "af60be3a9adfd20f013e8b13cd943561"
server_key_iv: "a77c94833575757901bc0544b20f3f4b"
server_key_version: "encryptionkey@0"
I20260925 15:03:20.701267 25432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:20.712615 25432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:20.716780 25432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:20.717058 25432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:20.735124 25432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34631
I20260925 15:03:20.735167 25498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34631 every 8 connection(s)
I20260925 15:03:20.736447 25432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:20.738252 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25432
I20260925 15:03:20.738440 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:20.739128 5071 external_mini_cluster.cc:1428] Setting key 854a9410b0f5f8252b14a139e7be1f4b
I20260925 15:03:20.741170 25499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:20.751685 25499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd: Bootstrap starting.
Sep 25 15:03:20 dist-test-slave-qxh0 krb5kdc[25419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:03:20.755122 25499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:20.756139 25499 log.cc:824] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:20.759382 25499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd: No bootstrap required, opened a new log
I20260925 15:03:20.760823 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:20.740898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48021 (local address 127.0.0.1:34631)
0925 15:03:20.741665 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:20.741676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:20.741715 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:03:20.743367 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:20.743386 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:20.743758 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:20.744141 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:20.745486 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:20.746852 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:20.748132 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:20.748806 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:20.749008 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:20.753270 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:20.753278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:20.753291 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:20.753334 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:20.756805 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:20.757621 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:20.757627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:20.757633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:20.757779 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:20.758247 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:20.758250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:20.758252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:20.758798 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:20.759016 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:20.759561 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 15:03:20.759877 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":370,"threads_started":1}
I20260925 15:03:20.764593 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:20.768707 25499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9897f28d44894ed68ac2cf650168c9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260925 15:03:20.769260 25499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:20.769379 25499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9897f28d44894ed68ac2cf650168c9bd, State: Initialized, Role: FOLLOWER
I20260925 15:03:20.770090 25499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9897f28d44894ed68ac2cf650168c9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260925 15:03:20.770349 25499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:20.770531 25499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:20.770736 25499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:20.773998 25499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9897f28d44894ed68ac2cf650168c9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260925 15:03:20.774595 25499 leader_election.cc:302] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9897f28d44894ed68ac2cf650168c9bd; no voters:
I20260925 15:03:20.775270 25499 leader_election.cc:288] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:20.775391 25509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:20.776732 25509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [term 1 LEADER]: Becoming Leader. State: Replica: 9897f28d44894ed68ac2cf650168c9bd, State: Running, Role: LEADER
I20260925 15:03:20.777242 25509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9897f28d44894ed68ac2cf650168c9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } }
I20260925 15:03:20.777508 25499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:20.779771 25511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9897f28d44894ed68ac2cf650168c9bd. Latest consensus state: current_term: 1 leader_uuid: "9897f28d44894ed68ac2cf650168c9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9897f28d44894ed68ac2cf650168c9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } } }
I20260925 15:03:20.780150 25511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:20.780722 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9897f28d44894ed68ac2cf650168c9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9897f28d44894ed68ac2cf650168c9bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34631 } } }
I20260925 15:03:20.781002 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:20.789146 25514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:20.791724 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:20.770422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35007 (local address 127.0.0.1:34631)
0925 15:03:20.770639 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:20.770647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:20.771216 (+ 569us) server_negotiation.cc:407] Connection header received
0925 15:03:20.771398 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:20.771401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:20.771463 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:20.771555 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:20.773369 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:20.774210 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:20.776045 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:20.776672 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:20.776792 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:20.780481 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:20.780491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:20.780499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:20.780550 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:20.785142 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:20.786473 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:20.786482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:20.786488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:20.786764 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:20.787481 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:20.787490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:20.787495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:20.788003 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:20.788385 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:20.790834 (+ 2449us) server_negotiation.cc:299] Negotiation successful
0925 15:03:20.791098 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:03:20.797910 25514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:20.806667 25514 catalog_manager.cc:1347] Generated new cluster ID: a82f5ebf07ac452fa27a5b8a70387893
I20260925 15:03:20.806782 25514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:20.847860 25514 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:20.849210 25514 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:20.858830 25514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9897f28d44894ed68ac2cf650168c9bd: Generated new TSK 0
I20260925 15:03:20.859678 25514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:20.869251 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25432
I20260925 15:03:20.871234 25494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:20.964178 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25432
W20260925 15:03:20.981508 25417 connection.cc:535] client connection to 127.0.0.1:34631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34631): Connection reset by peer (error 104)
2026-09-25T15:03:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:21 dist-test-slave-qxh0 krb5kdc[25532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:21 dist-test-slave-qxh0 krb5kdc[25532](info): set up 2 sockets
Sep 25 15:03:21 dist-test-slave-qxh0 krb5kdc[25532](info): commencing operation
krb5kdc: starting...
W20260925 15:03:22.059046 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.011s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:22 dist-test-slave-qxh0 krb5kdc[25532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:22.301949 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:22.623751 25545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:22.624019 25545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:22.624177 25545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:22.631359 25545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:22.631467 25545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:22.631546 25545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:22.631672 25545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:22.631738 25545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:22.641534 25545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:22.642870 25545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:22.644166 25545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:22.656114 25550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:22.656147 25551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:22.656203 25553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:22.657078 25545 server_base.cc:1034] running on GCE node
I20260925 15:03:22.657779 25545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:22.659144 25545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:22.660364 25545 hybrid_clock.cc:648] HybridClock initialized: now 1790348602660332 us; error 47 us; skew 500 ppm
I20260925 15:03:22.660715 25545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:22 dist-test-slave-qxh0 krb5kdc[25532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:22.663754 25545 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:22.665096 25545 webserver.cc:460] Webserver started at http://127.0.0.1:44043/ using document root <none> and password file <none>
I20260925 15:03:22.665702 25545 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:22.665833 25545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:22.666079 25545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:22.668920 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "2c14093a13f4487ca24666d2c3681114"
format_stamp: "Formatted at 2026-09-25 15:03:22 on dist-test-slave-qxh0"
server_key: "ebbc862819484b63be86aa0a4fdb59cc"
server_key_iv: "40f776617f6a0913a70edb67993b792e"
server_key_version: "encryptionkey@0"
I20260925 15:03:22.669591 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "2c14093a13f4487ca24666d2c3681114"
format_stamp: "Formatted at 2026-09-25 15:03:22 on dist-test-slave-qxh0"
server_key: "ebbc862819484b63be86aa0a4fdb59cc"
server_key_iv: "40f776617f6a0913a70edb67993b792e"
server_key_version: "encryptionkey@0"
I20260925 15:03:22.674461 25545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 15:03:22.677802 25560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:22.679183 25545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:22.679369 25545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "2c14093a13f4487ca24666d2c3681114"
format_stamp: "Formatted at 2026-09-25 15:03:22 on dist-test-slave-qxh0"
server_key: "ebbc862819484b63be86aa0a4fdb59cc"
server_key_iv: "40f776617f6a0913a70edb67993b792e"
server_key_version: "encryptionkey@0"
I20260925 15:03:22.679544 25545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:22.710150 25545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:22.714092 25545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:22.714372 25545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:22.732470 25545 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39741
I20260925 15:03:22.732494 25611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39741 every 8 connection(s)
I20260925 15:03:22.733809 25545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:22.734577 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25545
I20260925 15:03:22.734808 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:22.735378 5071 external_mini_cluster.cc:1428] Setting key c196ac023362614994ac802065f173e6
I20260925 15:03:22.738238 25612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:22.747932 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:22.737238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48419 (local address 127.0.0.1:39741)
0925 15:03:22.737978 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:22.737989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:22.738025 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:03:22.739666 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:22.739686 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:22.740041 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:22.740417 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:22.740939 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:22.742213 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:22.743622 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:22.744271 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:22.744455 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:22.745758 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:22.745765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:22.745778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:22.746338 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:22.746409 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:22.746432 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:22.746600 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:22.746911 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:03:22.747072 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":328,"threads_started":1}
I20260925 15:03:22.748739 25612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114: Bootstrap starting.
I20260925 15:03:22.751811 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:22.752391 25612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:22.753374 25612 log.cc:824] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:22.756237 25612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114: No bootstrap required, opened a new log
I20260925 15:03:22.762436 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:22.756254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40879 (local address 127.0.0.1:39741)
0925 15:03:22.756433 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:22.756442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:22.756980 (+ 538us) server_negotiation.cc:407] Connection header received
0925 15:03:22.757161 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:22.757165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:22.757220 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:22.757295 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:22.757864 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:22.758700 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:22.760016 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:22.760484 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:22.760617 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:22.761714 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:22.761718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:22.761721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:22.761744 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:22.761782 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:22.761799 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:22.761925 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:22.762128 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:03:22.762183 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:03:22.764911 25612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14093a13f4487ca24666d2c3681114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260925 15:03:22.765328 25612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:22.765444 25612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c14093a13f4487ca24666d2c3681114, State: Initialized, Role: FOLLOWER
I20260925 15:03:22.766031 25612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14093a13f4487ca24666d2c3681114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260925 15:03:22.766295 25612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:22.766439 25612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:22.766698 25612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:22.768970 25612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14093a13f4487ca24666d2c3681114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260925 15:03:22.769423 25612 leader_election.cc:302] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c14093a13f4487ca24666d2c3681114; no voters:
I20260925 15:03:22.769984 25612 leader_election.cc:288] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:22.770133 25622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:22.770829 25622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [term 1 LEADER]: Becoming Leader. State: Replica: 2c14093a13f4487ca24666d2c3681114, State: Running, Role: LEADER
I20260925 15:03:22.771391 25622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14093a13f4487ca24666d2c3681114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } }
I20260925 15:03:22.772321 25612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:22.774360 25623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c14093a13f4487ca24666d2c3681114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14093a13f4487ca24666d2c3681114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } } }
I20260925 15:03:22.774878 25623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:22.775557 25624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c14093a13f4487ca24666d2c3681114. Latest consensus state: current_term: 1 leader_uuid: "2c14093a13f4487ca24666d2c3681114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14093a13f4487ca24666d2c3681114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39741 } } }
I20260925 15:03:22.775848 25624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:22.777379 25630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:22.781889 25630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:22.790472 25630 catalog_manager.cc:1347] Generated new cluster ID: a32dee262daf466ab5dd4ed119b4307a
I20260925 15:03:22.790575 25630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:22.800815 25630 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:22.802711 25630 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:22.810894 25630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c14093a13f4487ca24666d2c3681114: Generated new TSK 0
I20260925 15:03:22.811878 25630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:22.821117 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25545
W20260925 15:03:22.822460 25607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48419
W20260925 15:03:22.823274 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:22.823459 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25545
W20260925 15:03:22.840270 25530 connection.cc:535] client connection to 127.0.0.1:39741 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39741): Connection reset by peer (error 104)
2026-09-25T15:03:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:22 dist-test-slave-qxh0 krb5kdc[25643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:22 dist-test-slave-qxh0 krb5kdc[25643](info): set up 2 sockets
Sep 25 15:03:22 dist-test-slave-qxh0 krb5kdc[25643](info): commencing operation
krb5kdc: starting...
W20260925 15:03:23.914933 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:24 dist-test-slave-qxh0 krb5kdc[25643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:24.157636 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:24.477037 25656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:24.477345 25656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:24.477525 25656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:24.484635 25656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:24.484740 25656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:24.484848 25656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:24.484958 25656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:24.485044 25656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:24.494817 25656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:24.496196 25656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:24.497501 25656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:24.508841 25661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:24.508833 25662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:24.508787 25664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:24.509613 25656 server_base.cc:1034] running on GCE node
I20260925 15:03:24.510371 25656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:24.511746 25656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:24.512981 25656 hybrid_clock.cc:648] HybridClock initialized: now 1790348604512939 us; error 51 us; skew 500 ppm
I20260925 15:03:24.513325 25656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:24 dist-test-slave-qxh0 krb5kdc[25643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:24.521982 25656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:24.523437 25656 webserver.cc:460] Webserver started at http://127.0.0.1:45777/ using document root <none> and password file <none>
I20260925 15:03:24.524091 25656 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:24.524184 25656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:24.524415 25656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:24.527295 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "e59935f30bfe45b8a43d3d0460dd8c78"
format_stamp: "Formatted at 2026-09-25 15:03:24 on dist-test-slave-qxh0"
server_key: "8e07ca0a873138a7fa43e5ac9c03d5ea"
server_key_iv: "9c316cc7a0ce18dff364554f11a4639d"
server_key_version: "encryptionkey@0"
I20260925 15:03:24.528024 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "e59935f30bfe45b8a43d3d0460dd8c78"
format_stamp: "Formatted at 2026-09-25 15:03:24 on dist-test-slave-qxh0"
server_key: "8e07ca0a873138a7fa43e5ac9c03d5ea"
server_key_iv: "9c316cc7a0ce18dff364554f11a4639d"
server_key_version: "encryptionkey@0"
I20260925 15:03:24.533322 25656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:03:24.536800 25671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:24.538151 25656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:03:24.538347 25656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "e59935f30bfe45b8a43d3d0460dd8c78"
format_stamp: "Formatted at 2026-09-25 15:03:24 on dist-test-slave-qxh0"
server_key: "8e07ca0a873138a7fa43e5ac9c03d5ea"
server_key_iv: "9c316cc7a0ce18dff364554f11a4639d"
server_key_version: "encryptionkey@0"
I20260925 15:03:24.538546 25656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:24.544203 25656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:24.548509 25656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:24.548800 25656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:24.566982 25656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34843
I20260925 15:03:24.567016 25722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34843 every 8 connection(s)
I20260925 15:03:24.568473 25656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:24.570469 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25656
I20260925 15:03:24.570705 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:24.571174 5071 external_mini_cluster.cc:1428] Setting key a42de020ad1b128dd069cf86b629ffc0
I20260925 15:03:24.572700 25723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:24.578955 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:24.572960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53113 (local address 127.0.0.1:34843)
0925 15:03:24.573726 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:24.573739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:24.573777 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:03:24.575465 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:24.575485 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:24.575836 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:24.576236 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:24.577271 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:24.577277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:24.577291 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:24.577304 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:24.577375 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:24.577394 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:24.577634 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:24.577789 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:03:24.577959 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":324,"threads_started":1}
I20260925 15:03:24.582404 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:24.584654 25723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78: Bootstrap starting.
I20260925 15:03:24.588389 25723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:24.589084 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:24.586487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35245 (local address 127.0.0.1:34843)
0925 15:03:24.586700 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:24.586708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:24.587205 (+ 497us) server_negotiation.cc:407] Connection header received
0925 15:03:24.587269 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:24.587273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:24.587361 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:24.587435 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:24.588414 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:24.588418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:24.588421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:24.588434 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:24.588469 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:24.588483 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:24.588617 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:24.588729 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 15:03:24.588776 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:03:24.589690 25723 log.cc:824] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:24.592303 25723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78: No bootstrap required, opened a new log
I20260925 15:03:24.599313 25723 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260925 15:03:24.599697 25723 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:24.599802 25723 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59935f30bfe45b8a43d3d0460dd8c78, State: Initialized, Role: FOLLOWER
I20260925 15:03:24.600402 25723 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260925 15:03:24.600631 25723 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:24.600768 25723 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:24.600951 25723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:24.603133 25723 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260925 15:03:24.603623 25723 leader_election.cc:302] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59935f30bfe45b8a43d3d0460dd8c78; no voters:
I20260925 15:03:24.604175 25723 leader_election.cc:288] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:24.604312 25733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:24.605005 25733 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [term 1 LEADER]: Becoming Leader. State: Replica: e59935f30bfe45b8a43d3d0460dd8c78, State: Running, Role: LEADER
I20260925 15:03:24.605518 25733 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } }
I20260925 15:03:24.606211 25723 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:24.608213 25734 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } } }
I20260925 15:03:24.608795 25734 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:24.609788 25735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59935f30bfe45b8a43d3d0460dd8c78. Latest consensus state: current_term: 1 leader_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59935f30bfe45b8a43d3d0460dd8c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34843 } } }
I20260925 15:03:24.611021 25735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:24.615547 25742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:24.621008 25742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:24.629699 25742 catalog_manager.cc:1347] Generated new cluster ID: 2f353f509afd43819309affe37bb20d4
I20260925 15:03:24.629820 25742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:24.647289 25742 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:24.647723 25742 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:24.653904 25742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59935f30bfe45b8a43d3d0460dd8c78: Generated new TSK 0
I20260925 15:03:24.654779 25742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:24.663075 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25656
W20260925 15:03:24.664283 25718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53113
W20260925 15:03:24.664999 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25656: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:24.665146 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25656
2026-09-25T15:03:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:24 dist-test-slave-qxh0 krb5kdc[25754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:24 dist-test-slave-qxh0 krb5kdc[25754](info): set up 2 sockets
Sep 25 15:03:24 dist-test-slave-qxh0 krb5kdc[25754](info): commencing operation
krb5kdc: starting...
W20260925 15:03:25.754925 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:25 dist-test-slave-qxh0 krb5kdc[25754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:26.002131 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:26.322922 25767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:26.323192 25767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:26.323294 25767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:26.330170 25767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:26.330282 25767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:26.330328 25767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:26.330387 25767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:26.330430 25767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:26.340118 25767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:26.341239 25767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:26.342470 25767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:26.353840 25773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:26.353809 25775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:26.354287 25772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:26.354792 25767 server_base.cc:1034] running on GCE node
I20260925 15:03:26.355475 25767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:26.356724 25767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:26.357941 25767 hybrid_clock.cc:648] HybridClock initialized: now 1790348606357890 us; error 47 us; skew 500 ppm
I20260925 15:03:26.358297 25767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:26 dist-test-slave-qxh0 krb5kdc[25754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:26.361369 25767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:26.362737 25767 webserver.cc:460] Webserver started at http://127.0.0.1:33053/ using document root <none> and password file <none>
I20260925 15:03:26.363344 25767 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:26.363446 25767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:26.363678 25767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:26.366523 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "3bfe886a182d4577ba70594be03c9497"
format_stamp: "Formatted at 2026-09-25 15:03:26 on dist-test-slave-qxh0"
server_key: "f0552f04dd1011ffd66972e709b0b242"
server_key_iv: "1e895fc1bdb2a8c6b80a8f2d97a70441"
server_key_version: "encryptionkey@0"
I20260925 15:03:26.367599 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "3bfe886a182d4577ba70594be03c9497"
format_stamp: "Formatted at 2026-09-25 15:03:26 on dist-test-slave-qxh0"
server_key: "f0552f04dd1011ffd66972e709b0b242"
server_key_iv: "1e895fc1bdb2a8c6b80a8f2d97a70441"
server_key_version: "encryptionkey@0"
I20260925 15:03:26.372684 25767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:26.376034 25782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:26.377292 25767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:03:26.377470 25767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "3bfe886a182d4577ba70594be03c9497"
format_stamp: "Formatted at 2026-09-25 15:03:26 on dist-test-slave-qxh0"
server_key: "f0552f04dd1011ffd66972e709b0b242"
server_key_iv: "1e895fc1bdb2a8c6b80a8f2d97a70441"
server_key_version: "encryptionkey@0"
I20260925 15:03:26.377643 25767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:26.393776 25767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:26.397569 25767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:26.397845 25767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:26.415490 25767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35177
I20260925 15:03:26.415513 25833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35177 every 8 connection(s)
I20260925 15:03:26.416844 25767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:26.420536 25834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:26.424932 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25767
I20260925 15:03:26.425132 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:26.425556 5071 external_mini_cluster.cc:1428] Setting key da7f052ef73a3bd5fc4358cd239a9868
I20260925 15:03:26.432578 25834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497: Bootstrap starting.
I20260925 15:03:26.435886 25834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:26.436971 25834 log.cc:824] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:26.439958 25834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497: No bootstrap required, opened a new log
Sep 25 15:03:26 dist-test-slave-qxh0 krb5kdc[25754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:03:26.450070 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:26.427397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48667 (local address 127.0.0.1:35177)
0925 15:03:26.428195 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:26.428216 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:26.428265 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:03:26.430839 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:26.430866 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:26.431347 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:26.431766 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:26.433402 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:26.434741 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:26.436007 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:26.436705 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:26.436898 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:26.441353 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:26.441363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:26.441381 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:26.441438 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:26.445447 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:26.446133 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:26.446142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:26.446150 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:26.446364 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:26.446771 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:26.446777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:26.446782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:26.447517 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:26.447822 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:26.448178 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 15:03:26.448634 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":344,"threads_started":1}
I20260925 15:03:26.450852 25834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfe886a182d4577ba70594be03c9497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260925 15:03:26.451256 25834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:26.451388 25834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfe886a182d4577ba70594be03c9497, State: Initialized, Role: FOLLOWER
I20260925 15:03:26.451987 25834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfe886a182d4577ba70594be03c9497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260925 15:03:26.452260 25834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:26.452418 25834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:26.452616 25834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:26.454372 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:26.455474 25834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfe886a182d4577ba70594be03c9497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260925 15:03:26.456053 25834 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bfe886a182d4577ba70594be03c9497; no voters:
I20260925 15:03:26.456650 25834 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:26.456773 25839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:26.458088 25839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [term 1 LEADER]: Becoming Leader. State: Replica: 3bfe886a182d4577ba70594be03c9497, State: Running, Role: LEADER
I20260925 15:03:26.458724 25839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfe886a182d4577ba70594be03c9497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } }
I20260925 15:03:26.459153 25834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:26.461431 25840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bfe886a182d4577ba70594be03c9497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfe886a182d4577ba70594be03c9497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } } }
I20260925 15:03:26.461455 25841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfe886a182d4577ba70594be03c9497. Latest consensus state: current_term: 1 leader_uuid: "3bfe886a182d4577ba70594be03c9497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfe886a182d4577ba70594be03c9497" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35177 } } }
I20260925 15:03:26.461890 25840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:26.461886 25841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:26.472018 25847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:26.477280 25847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:26.487264 25847 catalog_manager.cc:1347] Generated new cluster ID: 45b5cff9dd65411f80d155ffe3757e31
I20260925 15:03:26.487404 25847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:26.494050 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:26.475684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58447 (local address 127.0.0.1:35177)
0925 15:03:26.475935 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:26.475944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:26.476517 (+ 573us) server_negotiation.cc:407] Connection header received
0925 15:03:26.476865 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:26.476875 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:26.476981 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:26.477082 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:26.479786 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:26.481293 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:26.483249 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:26.483926 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:26.484035 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:26.487085 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:26.487092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:26.487098 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:26.487139 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:26.490802 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:26.491721 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:26.491726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:26.491730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:26.491857 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:26.492316 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:26.492321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:26.492324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:26.492686 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:26.492906 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:26.493327 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 15:03:26.493531 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 15:03:26.503257 25847 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:26.504420 25847 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:26.514384 25847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bfe886a182d4577ba70594be03c9497: Generated new TSK 0
I20260925 15:03:26.515275 25847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:26.526378 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25767
I20260925 15:03:26.527367 25829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:26.616866 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25767
2026-09-25T15:03:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:26 dist-test-slave-qxh0 krb5kdc[25866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:26 dist-test-slave-qxh0 krb5kdc[25866](info): set up 2 sockets
Sep 25 15:03:26 dist-test-slave-qxh0 krb5kdc[25866](info): commencing operation
krb5kdc: starting...
W20260925 15:03:27.702939 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:27 dist-test-slave-qxh0 krb5kdc[25866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:27.941380 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:28.261425 25879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:28.261693 25879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:28.261858 25879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:28.268915 25879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:28.269007 25879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:28.269117 25879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:28.269219 25879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:28.269301 25879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:28.278715 25879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:28.279894 25879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:28.281145 25879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:28.292577 25885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:28.292626 25887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:28.292658 25884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:28.293629 25879 server_base.cc:1034] running on GCE node
I20260925 15:03:28.294406 25879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:28.295930 25879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:28.297160 25879 hybrid_clock.cc:648] HybridClock initialized: now 1790348608297112 us; error 36 us; skew 500 ppm
I20260925 15:03:28.297547 25879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:28 dist-test-slave-qxh0 krb5kdc[25866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:28.300653 25879 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:28.301966 25879 webserver.cc:460] Webserver started at http://127.0.0.1:38973/ using document root <none> and password file <none>
I20260925 15:03:28.302703 25879 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:28.302812 25879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:28.303057 25879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:28.305958 25879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e"
format_stamp: "Formatted at 2026-09-25 15:03:28 on dist-test-slave-qxh0"
server_key: "9f8591e8abff0488eacbd48fb7d37170"
server_key_iv: "985256258172fe9499400e9b884fa481"
server_key_version: "encryptionkey@0"
I20260925 15:03:28.306706 25879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e"
format_stamp: "Formatted at 2026-09-25 15:03:28 on dist-test-slave-qxh0"
server_key: "9f8591e8abff0488eacbd48fb7d37170"
server_key_iv: "985256258172fe9499400e9b884fa481"
server_key_version: "encryptionkey@0"
I20260925 15:03:28.311558 25879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:03:28.314944 25894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:28.316241 25879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:03:28.316370 25879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e"
format_stamp: "Formatted at 2026-09-25 15:03:28 on dist-test-slave-qxh0"
server_key: "9f8591e8abff0488eacbd48fb7d37170"
server_key_iv: "985256258172fe9499400e9b884fa481"
server_key_version: "encryptionkey@0"
I20260925 15:03:28.316519 25879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:28.328117 25879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:28.331868 25879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:28.332104 25879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:28.349306 25879 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44989
I20260925 15:03:28.349344 25945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44989 every 8 connection(s)
I20260925 15:03:28.350675 25879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:28.353592 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25879
I20260925 15:03:28.353734 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:28.354159 5071 external_mini_cluster.cc:1428] Setting key b5afbbc281d52ea2c0e1fea59df95b5a
I20260925 15:03:28.354352 25946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:28.365958 25946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e: Bootstrap starting.
I20260925 15:03:28.366829 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:28.355875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51735 (local address 127.0.0.1:44989)
0925 15:03:28.356587 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:28.356598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:28.356635 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:03:28.358757 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:28.358777 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:28.359140 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:28.359540 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:28.360091 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:28.361325 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:28.362620 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:28.363291 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:28.363531 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:28.364641 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:28.364648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:28.364661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:28.365140 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:28.365208 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:28.365235 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:28.365371 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:28.365591 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:03:28.365840 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":324,"threads_started":1}
I20260925 15:03:28.369335 25946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:28.370417 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:28.370509 25946 log.cc:824] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:28.373219 25946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e: No bootstrap required, opened a new log
I20260925 15:03:28.381683 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:28.374778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52503 (local address 127.0.0.1:44989)
0925 15:03:28.374998 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:28.375005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:28.375457 (+ 452us) server_negotiation.cc:407] Connection header received
0925 15:03:28.375639 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:28.375642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:28.375707 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:28.375807 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:28.376361 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:28.377090 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:28.379002 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:28.379864 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:28.379967 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:28.381044 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:28.381049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:28.381052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:28.381068 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:28.381104 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:28.381119 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:28.381209 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:28.381366 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:03:28.381433 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:03:28.383598 25946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260925 15:03:28.383993 25946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:28.384097 25946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ea4afdc1bd4fe0b08eb5d510e31e2e, State: Initialized, Role: FOLLOWER
I20260925 15:03:28.384645 25946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260925 15:03:28.384878 25946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:28.385043 25946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:28.385214 25946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:28.387530 25946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260925 15:03:28.388083 25946 leader_election.cc:302] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ea4afdc1bd4fe0b08eb5d510e31e2e; no voters:
I20260925 15:03:28.388661 25946 leader_election.cc:288] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:28.388790 25956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:28.389514 25956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [term 1 LEADER]: Becoming Leader. State: Replica: 63ea4afdc1bd4fe0b08eb5d510e31e2e, State: Running, Role: LEADER
I20260925 15:03:28.390041 25956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260925 15:03:28.390828 25946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:28.392891 25958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ea4afdc1bd4fe0b08eb5d510e31e2e. Latest consensus state: current_term: 1 leader_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260925 15:03:28.392916 25957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea4afdc1bd4fe0b08eb5d510e31e2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260925 15:03:28.393348 25958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:28.393384 25957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:28.394208 25963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:28.399129 25963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:28.407089 25963 catalog_manager.cc:1347] Generated new cluster ID: 9c64131749fd46f4abd14bff1ab776d9
I20260925 15:03:28.407241 25963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:28.434257 25963 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:28.435565 25963 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:28.445648 25963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ea4afdc1bd4fe0b08eb5d510e31e2e: Generated new TSK 0
I20260925 15:03:28.446507 25963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:28.456517 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25879
W20260925 15:03:28.457643 25941 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51735
W20260925 15:03:28.458411 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25879: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:28.458564 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25879
2026-09-25T15:03:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:28 dist-test-slave-qxh0 krb5kdc[25977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:28 dist-test-slave-qxh0 krb5kdc[25977](info): set up 2 sockets
Sep 25 15:03:28 dist-test-slave-qxh0 krb5kdc[25977](info): commencing operation
krb5kdc: starting...
W20260925 15:03:29.542888 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:29 dist-test-slave-qxh0 krb5kdc[25977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:29.773370 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:30.093364 25990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:30.093716 25990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:30.093904 25990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:30.101085 25990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:30.101224 25990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:30.101321 25990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:30.101428 25990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:30.101507 25990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:30.110945 25990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:30.112147 25990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:30.113451 25990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:30.125496 25995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:30.125613 25996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:30.125492 25998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:30.126458 25990 server_base.cc:1034] running on GCE node
I20260925 15:03:30.127661 25990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:30.129731 25990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:30.131016 25990 hybrid_clock.cc:648] HybridClock initialized: now 1790348610130952 us; error 43 us; skew 500 ppm
I20260925 15:03:30.131524 25990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:30 dist-test-slave-qxh0 krb5kdc[25977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:30.135350 25990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:30.137538 25990 webserver.cc:460] Webserver started at http://127.0.0.1:45421/ using document root <none> and password file <none>
I20260925 15:03:30.138659 25990 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:30.138799 25990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:30.139243 25990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:30.143589 25990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "210708deb15847a2ad06b4419c27dbf6"
format_stamp: "Formatted at 2026-09-25 15:03:30 on dist-test-slave-qxh0"
server_key: "09b06073fd6f25976121a4fe7e46db4e"
server_key_iv: "acc28c72d5fa5d07a0dad26477731e2f"
server_key_version: "encryptionkey@0"
I20260925 15:03:30.144623 25990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "210708deb15847a2ad06b4419c27dbf6"
format_stamp: "Formatted at 2026-09-25 15:03:30 on dist-test-slave-qxh0"
server_key: "09b06073fd6f25976121a4fe7e46db4e"
server_key_iv: "acc28c72d5fa5d07a0dad26477731e2f"
server_key_version: "encryptionkey@0"
I20260925 15:03:30.151474 25990 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260925 15:03:30.156131 26005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:30.157857 25990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:03:30.158047 25990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "210708deb15847a2ad06b4419c27dbf6"
format_stamp: "Formatted at 2026-09-25 15:03:30 on dist-test-slave-qxh0"
server_key: "09b06073fd6f25976121a4fe7e46db4e"
server_key_iv: "acc28c72d5fa5d07a0dad26477731e2f"
server_key_version: "encryptionkey@0"
I20260925 15:03:30.158305 25990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:30.164256 25990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:30.169471 25990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:30.169744 25990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:30.188156 25990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41859
I20260925 15:03:30.188205 26056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41859 every 8 connection(s)
I20260925 15:03:30.189564 25990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:30.193300 26057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:30.194968 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 25990
I20260925 15:03:30.195147 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:30.195585 5071 external_mini_cluster.cc:1428] Setting key 239a4a59d7450fbd4b0b8ed4546cf164
I20260925 15:03:30.203686 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:30.197497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54215 (local address 127.0.0.1:41859)
0925 15:03:30.198279 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:30.198293 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:30.198340 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:03:30.199929 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:30.199973 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:30.200433 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:30.200888 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:30.201799 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:30.201807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:30.201823 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:30.201838 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:30.201904 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:30.201928 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:30.202134 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:30.202266 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:03:30.202447 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":321,"threads_started":1}
I20260925 15:03:30.207726 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:30.208037 26057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6: Bootstrap starting.
I20260925 15:03:30.211956 26057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:30.213255 26057 log.cc:824] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:30.215761 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:30.212932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40711 (local address 127.0.0.1:41859)
0925 15:03:30.213138 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:30.213144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:30.213653 (+ 509us) server_negotiation.cc:407] Connection header received
0925 15:03:30.213829 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:30.213837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:30.213923 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:30.214048 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:30.215090 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:30.215095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:30.215099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:30.215117 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:30.215161 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:30.215180 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:30.215322 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:30.215418 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 15:03:30.215475 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:03:30.216682 26057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6: No bootstrap required, opened a new log
I20260925 15:03:30.223840 26057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210708deb15847a2ad06b4419c27dbf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260925 15:03:30.224207 26057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:30.224318 26057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210708deb15847a2ad06b4419c27dbf6, State: Initialized, Role: FOLLOWER
I20260925 15:03:30.224870 26057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210708deb15847a2ad06b4419c27dbf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260925 15:03:30.225116 26057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:30.225256 26057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:30.225421 26057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:30.227617 26057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210708deb15847a2ad06b4419c27dbf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260925 15:03:30.228091 26057 leader_election.cc:302] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210708deb15847a2ad06b4419c27dbf6; no voters:
I20260925 15:03:30.228614 26057 leader_election.cc:288] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:30.228737 26067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:30.229398 26067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [term 1 LEADER]: Becoming Leader. State: Replica: 210708deb15847a2ad06b4419c27dbf6, State: Running, Role: LEADER
I20260925 15:03:30.230934 26057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:30.231050 26067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210708deb15847a2ad06b4419c27dbf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } }
I20260925 15:03:30.233603 26068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210708deb15847a2ad06b4419c27dbf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210708deb15847a2ad06b4419c27dbf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260925 15:03:30.233990 26068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:30.234479 26069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210708deb15847a2ad06b4419c27dbf6. Latest consensus state: current_term: 1 leader_uuid: "210708deb15847a2ad06b4419c27dbf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210708deb15847a2ad06b4419c27dbf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41859 } } }
I20260925 15:03:30.234794 26069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:30.239066 26074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:30.244200 26074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:30.253067 26074 catalog_manager.cc:1347] Generated new cluster ID: e64e8c2c54e2446cb8d8174982acac54
I20260925 15:03:30.253204 26074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:30.269428 26074 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:30.269907 26074 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:30.291136 26074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 210708deb15847a2ad06b4419c27dbf6: Generated new TSK 0
I20260925 15:03:30.291982 26074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:30.299703 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25990
W20260925 15:03:30.300866 26052 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54215
W20260925 15:03:30.301568 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 25990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:30.301712 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 25990
2026-09-25T15:03:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:30 dist-test-slave-qxh0 krb5kdc[26088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:30 dist-test-slave-qxh0 krb5kdc[26088](info): set up 2 sockets
Sep 25 15:03:30 dist-test-slave-qxh0 krb5kdc[26088](info): commencing operation
krb5kdc: starting...
W20260925 15:03:31.383025 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:31 dist-test-slave-qxh0 krb5kdc[26088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:31.622694 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:31.949887 26101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:31.950148 26101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:31.950237 26101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:31.957803 26101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:31.957903 26101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:31.957948 26101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:31.958006 26101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:31.958047 26101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:31.967590 26101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:31.968696 26101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:31.969877 26101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:31.981974 26106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:31.982059 26107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:31.982090 26109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:31.983121 26101 server_base.cc:1034] running on GCE node
I20260925 15:03:31.983826 26101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:31.985157 26101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:31.986377 26101 hybrid_clock.cc:648] HybridClock initialized: now 1790348611986341 us; error 47 us; skew 500 ppm
I20260925 15:03:31.986763 26101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:31 dist-test-slave-qxh0 krb5kdc[26088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:31.989638 26101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:31.991029 26101 webserver.cc:460] Webserver started at http://127.0.0.1:40471/ using document root <none> and password file <none>
I20260925 15:03:31.991665 26101 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:31.991804 26101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:31.992091 26101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:31.994968 26101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "3aec0301a9ef42988e876f92ee559a20"
format_stamp: "Formatted at 2026-09-25 15:03:31 on dist-test-slave-qxh0"
server_key: "2f5655351e5e918f5df1379cdf743e72"
server_key_iv: "261fbe403f4e4bffc00fd7b4995cb2e1"
server_key_version: "encryptionkey@0"
I20260925 15:03:31.995702 26101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "3aec0301a9ef42988e876f92ee559a20"
format_stamp: "Formatted at 2026-09-25 15:03:31 on dist-test-slave-qxh0"
server_key: "2f5655351e5e918f5df1379cdf743e72"
server_key_iv: "261fbe403f4e4bffc00fd7b4995cb2e1"
server_key_version: "encryptionkey@0"
I20260925 15:03:32.000437 26101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:03:32.003609 26116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:32.004809 26101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:32.004982 26101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "3aec0301a9ef42988e876f92ee559a20"
format_stamp: "Formatted at 2026-09-25 15:03:31 on dist-test-slave-qxh0"
server_key: "2f5655351e5e918f5df1379cdf743e72"
server_key_iv: "261fbe403f4e4bffc00fd7b4995cb2e1"
server_key_version: "encryptionkey@0"
I20260925 15:03:32.005158 26101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:32.019165 26101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:32.023010 26101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:32.023296 26101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:32.040889 26101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43327
I20260925 15:03:32.040923 26167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43327 every 8 connection(s)
I20260925 15:03:32.042133 26101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:32.045454 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26101
I20260925 15:03:32.045616 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:32.045881 26168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:32.046217 5071 external_mini_cluster.cc:1428] Setting key 057c7f1f3474bba577db1db6f55e1458
I20260925 15:03:32.060215 26168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20: Bootstrap starting.
Sep 25 15:03:32 dist-test-slave-qxh0 krb5kdc[26088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:03:32.063398 26168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:32.064446 26168 log.cc:824] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:32.067188 26168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20: No bootstrap required, opened a new log
I20260925 15:03:32.068660 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:32.047880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:52733 (local address 127.0.0.1:43327)
0925 15:03:32.048618 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:32.048628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:32.048670 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:03:32.050333 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:32.050365 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:32.050770 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:32.051053 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:32.052712 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:32.053959 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:32.055826 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:32.056525 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:32.056679 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:32.061527 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:32.061537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:32.061551 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:32.061586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:32.064727 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:32.065490 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:32.065497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:32.065503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:32.065652 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:32.066136 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:32.066143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:32.066151 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:32.066686 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:32.066997 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:32.067464 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:03:32.067783 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":343,"threads_started":1}
I20260925 15:03:32.072518 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:32.075748 26168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aec0301a9ef42988e876f92ee559a20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260925 15:03:32.076119 26168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:32.076224 26168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aec0301a9ef42988e876f92ee559a20, State: Initialized, Role: FOLLOWER
I20260925 15:03:32.076905 26168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aec0301a9ef42988e876f92ee559a20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260925 15:03:32.077220 26168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:32.077375 26168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:32.077570 26168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:32.079874 26168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aec0301a9ef42988e876f92ee559a20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260925 15:03:32.080366 26168 leader_election.cc:302] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aec0301a9ef42988e876f92ee559a20; no voters:
I20260925 15:03:32.081122 26168 leader_election.cc:288] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:32.081218 26178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:32.082378 26178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [term 1 LEADER]: Becoming Leader. State: Replica: 3aec0301a9ef42988e876f92ee559a20, State: Running, Role: LEADER
I20260925 15:03:32.083011 26178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aec0301a9ef42988e876f92ee559a20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260925 15:03:32.083552 26168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:32.085794 26180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aec0301a9ef42988e876f92ee559a20. Latest consensus state: current_term: 1 leader_uuid: "3aec0301a9ef42988e876f92ee559a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aec0301a9ef42988e876f92ee559a20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } } }
I20260925 15:03:32.086488 26180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:32.086053 26179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aec0301a9ef42988e876f92ee559a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aec0301a9ef42988e876f92ee559a20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } } }
I20260925 15:03:32.086968 26179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:32.089777 26186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:32.092585 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:32.076938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:37943 (local address 127.0.0.1:43327)
0925 15:03:32.077115 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:32.077121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:32.077401 (+ 280us) server_negotiation.cc:407] Connection header received
0925 15:03:32.077592 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:32.077595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:32.077655 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:32.077760 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:32.079215 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:32.079949 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:32.081380 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:32.082095 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:32.082208 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:32.085190 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:32.085198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:32.085205 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:32.085252 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:32.089560 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:32.090331 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:32.090336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:32.090339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:32.090454 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:32.091011 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:32.091018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:32.091024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:32.091406 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:32.091687 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:32.092048 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 15:03:32.092204 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:03:32.097977 26186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:32.106772 26186 catalog_manager.cc:1347] Generated new cluster ID: 31cf9ab14a0248608d65d524fab0867b
I20260925 15:03:32.106890 26186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:32.125088 26186 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:32.126746 26186 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:32.144299 26186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aec0301a9ef42988e876f92ee559a20: Generated new TSK 0
I20260925 15:03:32.145218 26186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:32.159999 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26101
I20260925 15:03:32.161037 26163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:32.252188 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26101
W20260925 15:03:32.268996 26086 connection.cc:535] client connection to 127.0.0.1:43327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43327): Connection reset by peer (error 104)
2026-09-25T15:03:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:32 dist-test-slave-qxh0 krb5kdc[26200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:32 dist-test-slave-qxh0 krb5kdc[26200](info): set up 2 sockets
Sep 25 15:03:32 dist-test-slave-qxh0 krb5kdc[26200](info): commencing operation
krb5kdc: starting...
W20260925 15:03:33.347049 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:33 dist-test-slave-qxh0 krb5kdc[26200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:33.589623 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:33.906066 26213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:33.906383 26213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:33.906549 26213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:33.913421 26213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:33.913568 26213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:33.913671 26213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:33.913774 26213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:33.913867 26213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:33.926875 26213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:33.928122 26213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:33.929340 26213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:33.941036 26218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:33.941066 26221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:33.940955 26219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:33.941614 26213 server_base.cc:1034] running on GCE node
I20260925 15:03:33.942562 26213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:33.944039 26213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:33.945278 26213 hybrid_clock.cc:648] HybridClock initialized: now 1790348613945240 us; error 35 us; skew 500 ppm
I20260925 15:03:33.945624 26213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:33 dist-test-slave-qxh0 krb5kdc[26200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:33.948764 26213 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:33.950220 26213 webserver.cc:460] Webserver started at http://127.0.0.1:35769/ using document root <none> and password file <none>
I20260925 15:03:33.951025 26213 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:33.951179 26213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:33.951485 26213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:33.954406 26213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "fc464c01fcb849749a3ba997dcd558c0"
format_stamp: "Formatted at 2026-09-25 15:03:33 on dist-test-slave-qxh0"
server_key: "339bd58da8ccfc5a82280123c3290800"
server_key_iv: "30d49d4aa55d6826a57cb505cd94155e"
server_key_version: "encryptionkey@0"
I20260925 15:03:33.955214 26213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "fc464c01fcb849749a3ba997dcd558c0"
format_stamp: "Formatted at 2026-09-25 15:03:33 on dist-test-slave-qxh0"
server_key: "339bd58da8ccfc5a82280123c3290800"
server_key_iv: "30d49d4aa55d6826a57cb505cd94155e"
server_key_version: "encryptionkey@0"
I20260925 15:03:33.960016 26213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:03:33.963270 26228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:33.964561 26213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:33.964738 26213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "fc464c01fcb849749a3ba997dcd558c0"
format_stamp: "Formatted at 2026-09-25 15:03:33 on dist-test-slave-qxh0"
server_key: "339bd58da8ccfc5a82280123c3290800"
server_key_iv: "30d49d4aa55d6826a57cb505cd94155e"
server_key_version: "encryptionkey@0"
I20260925 15:03:33.964931 26213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:33.981981 26213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:33.986516 26213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:33.986801 26213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:34.004820 26213 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46667
I20260925 15:03:34.004899 26279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46667 every 8 connection(s)
I20260925 15:03:34.006157 26213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:34.009781 26280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:34.011207 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26213
I20260925 15:03:34.011400 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:34.011849 5071 external_mini_cluster.cc:1428] Setting key 19b1ffa782e6d670a8022b09e903222a
I20260925 15:03:34.024356 26280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0: Bootstrap starting.
I20260925 15:03:34.026091 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:34.013496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:60537 (local address 127.0.0.1:46667)
0925 15:03:34.014280 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:34.014295 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:34.014342 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:03:34.016245 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:34.016274 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:34.016828 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:34.017286 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:34.017799 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:34.019686 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:34.021020 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:34.022084 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:34.022247 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:34.023393 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:34.023402 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:34.023420 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:34.024125 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:34.024221 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:34.024250 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:34.024428 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:34.024670 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:03:34.024851 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":350,"threads_started":1}
I20260925 15:03:34.027889 26280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:34.028985 26280 log.cc:824] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:34.029928 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:34.031894 26280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0: No bootstrap required, opened a new log
I20260925 15:03:34.041043 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:34.034794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:58429 (local address 127.0.0.1:46667)
0925 15:03:34.034981 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:34.034987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:34.035372 (+ 385us) server_negotiation.cc:407] Connection header received
0925 15:03:34.035497 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:34.035502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:34.035578 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:34.035690 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:34.036114 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:34.037213 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:34.038476 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:34.039214 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:34.039347 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:34.040356 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:34.040361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:34.040365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:34.040388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:34.040429 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:34.040447 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:34.040564 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:34.040707 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 15:03:34.040752 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:03:34.041437 26280 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc464c01fcb849749a3ba997dcd558c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260925 15:03:34.041779 26280 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:34.041893 26280 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc464c01fcb849749a3ba997dcd558c0, State: Initialized, Role: FOLLOWER
I20260925 15:03:34.042462 26280 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc464c01fcb849749a3ba997dcd558c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260925 15:03:34.042747 26280 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:34.042881 26280 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:34.043057 26280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:34.045053 26280 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc464c01fcb849749a3ba997dcd558c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260925 15:03:34.045495 26280 leader_election.cc:302] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc464c01fcb849749a3ba997dcd558c0; no voters:
I20260925 15:03:34.046031 26280 leader_election.cc:288] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:34.046151 26290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:34.047402 26290 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [term 1 LEADER]: Becoming Leader. State: Replica: fc464c01fcb849749a3ba997dcd558c0, State: Running, Role: LEADER
I20260925 15:03:34.048095 26280 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:34.048019 26290 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc464c01fcb849749a3ba997dcd558c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } }
I20260925 15:03:34.050563 26292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc464c01fcb849749a3ba997dcd558c0. Latest consensus state: current_term: 1 leader_uuid: "fc464c01fcb849749a3ba997dcd558c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc464c01fcb849749a3ba997dcd558c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } } }
I20260925 15:03:34.050776 26291 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc464c01fcb849749a3ba997dcd558c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc464c01fcb849749a3ba997dcd558c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46667 } } }
I20260925 15:03:34.051031 26292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:34.051036 26291 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:34.051851 26298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:34.059368 26298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:34.072546 26298 catalog_manager.cc:1347] Generated new cluster ID: b6a8b612d9d34ac4850727d3ce2f4a8c
I20260925 15:03:34.072682 26298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:34.099798 26298 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:34.101137 26298 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:34.108572 26298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc464c01fcb849749a3ba997dcd558c0: Generated new TSK 0
I20260925 15:03:34.109364 26298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:34.117090 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26213
W20260925 15:03:34.118386 26275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:60537
W20260925 15:03:34.119109 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26213: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:34.119248 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26213
W20260925 15:03:34.135906 26198 connection.cc:535] client connection to 127.0.0.1:46667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46667): Connection reset by peer (error 104)
2026-09-25T15:03:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1866 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:34 dist-test-slave-qxh0 krb5kdc[26311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:34 dist-test-slave-qxh0 krb5kdc[26311](info): set up 2 sockets
Sep 25 15:03:34 dist-test-slave-qxh0 krb5kdc[26311](info): commencing operation
krb5kdc: starting...
W20260925 15:03:35.199012 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:35 dist-test-slave-qxh0 krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:35.450394 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:35.766278 26324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:35.766538 26324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:35.766700 26324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:35.773527 26324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:35.773623 26324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:35.773667 26324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:35.773726 26324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:35.773763 26324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:35.783321 26324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:35.784502 26324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:35.785702 26324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:35.797361 26329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:35.797391 26330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:35.797643 26332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:35.798344 26324 server_base.cc:1034] running on GCE node
I20260925 15:03:35.799100 26324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:35.800446 26324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:35.801652 26324 hybrid_clock.cc:648] HybridClock initialized: now 1790348615801593 us; error 48 us; skew 500 ppm
I20260925 15:03:35.802004 26324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:35 dist-test-slave-qxh0 krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:35.808670 26324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:35.810141 26324 webserver.cc:460] Webserver started at http://127.0.0.1:38111/ using document root <none> and password file <none>
I20260925 15:03:35.810889 26324 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:35.811018 26324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:35.811333 26324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:35.814182 26324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "87398525bc5f4abb8d4faf9cd0e0ff50"
format_stamp: "Formatted at 2026-09-25 15:03:35 on dist-test-slave-qxh0"
server_key: "fabd8bb88d2f58a31267c8e93c5a628e"
server_key_iv: "8d7e6ce9bbe3f1ec706f17a7d067b5ec"
server_key_version: "encryptionkey@0"
I20260925 15:03:35.814946 26324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "87398525bc5f4abb8d4faf9cd0e0ff50"
format_stamp: "Formatted at 2026-09-25 15:03:35 on dist-test-slave-qxh0"
server_key: "fabd8bb88d2f58a31267c8e93c5a628e"
server_key_iv: "8d7e6ce9bbe3f1ec706f17a7d067b5ec"
server_key_version: "encryptionkey@0"
I20260925 15:03:35.820117 26324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:03:35.823511 26339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:35.824709 26324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:03:35.824892 26324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "87398525bc5f4abb8d4faf9cd0e0ff50"
format_stamp: "Formatted at 2026-09-25 15:03:35 on dist-test-slave-qxh0"
server_key: "fabd8bb88d2f58a31267c8e93c5a628e"
server_key_iv: "8d7e6ce9bbe3f1ec706f17a7d067b5ec"
server_key_version: "encryptionkey@0"
I20260925 15:03:35.825084 26324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:35.830639 26324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:35.834337 26324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:35.834618 26324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:35.854265 26324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44073
I20260925 15:03:35.854318 26390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44073 every 8 connection(s)
I20260925 15:03:35.855733 26324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:35.859464 26391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:35.863932 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26324
I20260925 15:03:35.864125 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:35.864675 5071 external_mini_cluster.cc:1428] Setting key d097a192a7057289384de2c3167048a4
I20260925 15:03:35.870677 26391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50: Bootstrap starting.
I20260925 15:03:35.872467 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:35.866385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:51471 (local address 127.0.0.1:44073)
0925 15:03:35.867186 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:35.867197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:35.867240 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:03:35.869132 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:35.869153 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:35.869456 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:35.869740 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:35.870898 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:35.870907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:35.870930 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:35.870946 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:35.871006 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:35.871030 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:35.871207 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:35.871384 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:03:35.871588 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":411,"threads_started":1}
I20260925 15:03:35.873950 26391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:35.875146 26391 log.cc:824] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:35.875986 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:35.878037 26391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50: No bootstrap required, opened a new log
I20260925 15:03:35.883291 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:35.880928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:53903 (local address 127.0.0.1:44073)
0925 15:03:35.881107 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:35.881113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:35.881501 (+ 388us) server_negotiation.cc:407] Connection header received
0925 15:03:35.881621 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:35.881627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:35.881681 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:35.881797 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:35.882577 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:35.882582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:35.882586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:35.882648 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:35.882689 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:35.882713 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:35.882843 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:35.882924 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 15:03:35.882981 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:03:35.886672 26391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260925 15:03:35.887048 26391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:35.887163 26391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87398525bc5f4abb8d4faf9cd0e0ff50, State: Initialized, Role: FOLLOWER
I20260925 15:03:35.887745 26391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260925 15:03:35.887995 26391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:35.888144 26391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:35.888314 26391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:35.890631 26391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260925 15:03:35.891125 26391 leader_election.cc:302] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87398525bc5f4abb8d4faf9cd0e0ff50; no voters:
I20260925 15:03:35.891664 26391 leader_election.cc:288] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:35.891835 26401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:35.892488 26401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [term 1 LEADER]: Becoming Leader. State: Replica: 87398525bc5f4abb8d4faf9cd0e0ff50, State: Running, Role: LEADER
I20260925 15:03:35.893000 26401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260925 15:03:35.893584 26391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:35.896004 26403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87398525bc5f4abb8d4faf9cd0e0ff50. Latest consensus state: current_term: 1 leader_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260925 15:03:35.896102 26402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87398525bc5f4abb8d4faf9cd0e0ff50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260925 15:03:35.896435 26403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:35.896492 26402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:35.898056 26406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:35.903177 26406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:35.911698 26406 catalog_manager.cc:1347] Generated new cluster ID: 82047da714ac4de7bf0fbd3d81cbd689
I20260925 15:03:35.911787 26406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:35.940544 26406 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:35.940963 26406 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:35.947081 26406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87398525bc5f4abb8d4faf9cd0e0ff50: Generated new TSK 0
I20260925 15:03:35.947850 26406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:35.961421 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26324
W20260925 15:03:35.962409 26386 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:51471
W20260925 15:03:35.963028 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26324: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:35.963174 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26324
2026-09-25T15:03:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1844 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:36 dist-test-slave-qxh0 krb5kdc[26422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:36 dist-test-slave-qxh0 krb5kdc[26422](info): set up 2 sockets
Sep 25 15:03:36 dist-test-slave-qxh0 krb5kdc[26422](info): commencing operation
krb5kdc: starting...
W20260925 15:03:37.058909 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:37 dist-test-slave-qxh0 krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:37.299741 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:37.624084 26435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:37.624372 26435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:37.624467 26435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:37.631528 26435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:37.631632 26435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:37.631677 26435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:37.631737 26435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:37.631780 26435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:37.644202 26435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:37.645417 26435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:37.646721 26435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:37.657758 26440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:37.657716 26443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:37.657878 26441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:37.658926 26435 server_base.cc:1034] running on GCE node
I20260925 15:03:37.659650 26435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:37.661149 26435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:37.662400 26435 hybrid_clock.cc:648] HybridClock initialized: now 1790348617662354 us; error 55 us; skew 500 ppm
I20260925 15:03:37.662824 26435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:37 dist-test-slave-qxh0 krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:37.665974 26435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:37.667475 26435 webserver.cc:460] Webserver started at http://127.0.0.1:46847/ using document root <none> and password file <none>
I20260925 15:03:37.668097 26435 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:37.668198 26435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:37.668437 26435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:37.671413 26435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "18ccaa1a22c244c688c67eec93d6ca48"
format_stamp: "Formatted at 2026-09-25 15:03:37 on dist-test-slave-qxh0"
server_key: "a47b1e61bf27beffa6ee1e42cd470308"
server_key_iv: "611da91f86a5a92b2162da69111e2799"
server_key_version: "encryptionkey@0"
I20260925 15:03:37.672092 26435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "18ccaa1a22c244c688c67eec93d6ca48"
format_stamp: "Formatted at 2026-09-25 15:03:37 on dist-test-slave-qxh0"
server_key: "a47b1e61bf27beffa6ee1e42cd470308"
server_key_iv: "611da91f86a5a92b2162da69111e2799"
server_key_version: "encryptionkey@0"
I20260925 15:03:37.677090 26435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:37.680523 26450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:37.681829 26435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 15:03:37.682000 26435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "18ccaa1a22c244c688c67eec93d6ca48"
format_stamp: "Formatted at 2026-09-25 15:03:37 on dist-test-slave-qxh0"
server_key: "a47b1e61bf27beffa6ee1e42cd470308"
server_key_iv: "611da91f86a5a92b2162da69111e2799"
server_key_version: "encryptionkey@0"
I20260925 15:03:37.682173 26435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:37.717535 26435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:37.721946 26435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:37.722215 26435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:37.740229 26435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40977
I20260925 15:03:37.740304 26501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40977 every 8 connection(s)
I20260925 15:03:37.741631 26435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:37.742475 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26435
I20260925 15:03:37.742695 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:37.743149 5071 external_mini_cluster.cc:1428] Setting key 8e51344b950d94d58cc43468e76d2922
I20260925 15:03:37.746382 26502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:03:37 dist-test-slave-qxh0 krb5kdc[26422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:03:37.757028 26502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48: Bootstrap starting.
I20260925 15:03:37.760181 26502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:37.761178 26502 log.cc:824] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:37.764318 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:37.744943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:43263 (local address 127.0.0.1:40977)
0925 15:03:37.745712 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:37.745724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:37.745776 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:03:37.747587 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:37.747626 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:37.747992 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:37.748370 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:37.749738 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:37.751132 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:37.752334 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:37.752991 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:37.753187 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:37.757170 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:37.757179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:37.757197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:37.757243 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:37.760066 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:37.760842 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:37.760849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:37.760854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:37.761002 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:37.761457 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:37.761461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:37.761463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:37.762187 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:37.762502 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:37.763044 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 15:03:37.763265 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":332,"threads_started":1}
I20260925 15:03:37.764297 26502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48: No bootstrap required, opened a new log
I20260925 15:03:37.768251 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:37.775352 26502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ccaa1a22c244c688c67eec93d6ca48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260925 15:03:37.775705 26502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:37.775837 26502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ccaa1a22c244c688c67eec93d6ca48, State: Initialized, Role: FOLLOWER
I20260925 15:03:37.776429 26502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ccaa1a22c244c688c67eec93d6ca48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260925 15:03:37.776685 26502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:37.776845 26502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:37.777034 26502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:37.779386 26502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ccaa1a22c244c688c67eec93d6ca48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260925 15:03:37.779927 26502 leader_election.cc:302] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18ccaa1a22c244c688c67eec93d6ca48; no voters:
I20260925 15:03:37.780619 26502 leader_election.cc:288] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:37.780721 26512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:37.782019 26512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [term 1 LEADER]: Becoming Leader. State: Replica: 18ccaa1a22c244c688c67eec93d6ca48, State: Running, Role: LEADER
I20260925 15:03:37.782724 26512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ccaa1a22c244c688c67eec93d6ca48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260925 15:03:37.783229 26502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:37.785701 26514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ccaa1a22c244c688c67eec93d6ca48. Latest consensus state: current_term: 1 leader_uuid: "18ccaa1a22c244c688c67eec93d6ca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ccaa1a22c244c688c67eec93d6ca48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260925 15:03:37.786088 26514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:37.786677 26513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18ccaa1a22c244c688c67eec93d6ca48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ccaa1a22c244c688c67eec93d6ca48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260925 15:03:37.786974 26513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:37.790809 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:37.773725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:36757 (local address 127.0.0.1:40977)
0925 15:03:37.773925 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:37.773932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:37.774330 (+ 398us) server_negotiation.cc:407] Connection header received
0925 15:03:37.774468 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:37.774473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:37.774554 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:37.774677 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:37.776045 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:37.776983 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:37.778198 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:37.778724 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:37.778849 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:37.781520 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:37.781526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:37.781530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:37.781570 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:37.785087 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:37.786242 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:37.786249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:37.786254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:37.786513 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:37.787239 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:37.787247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:37.787252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:37.787732 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:37.788114 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:37.788874 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 15:03:37.789138 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:03:37.792636 26519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:37.798000 26519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:37.807185 26519 catalog_manager.cc:1347] Generated new cluster ID: 025f4eb6c94340b1a9f21697eaa062de
I20260925 15:03:37.807343 26519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:37.815090 26519 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:37.816742 26519 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:37.827047 26519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18ccaa1a22c244c688c67eec93d6ca48: Generated new TSK 0
I20260925 15:03:37.828025 26519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:37.839828 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26435
I20260925 15:03:37.840818 26497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:37.933097 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26435
W20260925 15:03:37.950049 26420 connection.cc:535] client connection to 127.0.0.1:40977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40977): Connection reset by peer (error 104)
2026-09-25T15:03:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:38 dist-test-slave-qxh0 krb5kdc[26534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:38 dist-test-slave-qxh0 krb5kdc[26534](info): set up 2 sockets
Sep 25 15:03:38 dist-test-slave-qxh0 krb5kdc[26534](info): commencing operation
krb5kdc: starting...
W20260925 15:03:39.027091 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:39 dist-test-slave-qxh0 krb5kdc[26534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:39.268453 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:39.590510 26547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:39.590811 26547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:39.591030 26547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:39.598234 26547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:39.598333 26547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:39.598464 26547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:39.598570 26547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:39.598647 26547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:39.608134 26547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:39.609360 26547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:39.610675 26547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:39.622102 26552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:39.622049 26555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:39.622012 26553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:39.622982 26547 server_base.cc:1034] running on GCE node
I20260925 15:03:39.623800 26547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:39.625264 26547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:39.626498 26547 hybrid_clock.cc:648] HybridClock initialized: now 1790348619626455 us; error 61 us; skew 500 ppm
I20260925 15:03:39.626871 26547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:39 dist-test-slave-qxh0 krb5kdc[26534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:39.629967 26547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:39.631479 26547 webserver.cc:460] Webserver started at http://127.0.0.1:41195/ using document root <none> and password file <none>
I20260925 15:03:39.632102 26547 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:39.632195 26547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:39.632428 26547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:39.635339 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "797bcd802d864865aa8b3d489ee7cae3"
format_stamp: "Formatted at 2026-09-25 15:03:39 on dist-test-slave-qxh0"
server_key: "a3bc93253bb45566727d8cf4d4feed0c"
server_key_iv: "2f53c16ba919cddc4cbec114cb69285f"
server_key_version: "encryptionkey@0"
I20260925 15:03:39.636022 26547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "797bcd802d864865aa8b3d489ee7cae3"
format_stamp: "Formatted at 2026-09-25 15:03:39 on dist-test-slave-qxh0"
server_key: "a3bc93253bb45566727d8cf4d4feed0c"
server_key_iv: "2f53c16ba919cddc4cbec114cb69285f"
server_key_version: "encryptionkey@0"
I20260925 15:03:39.640723 26547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:03:39.643900 26562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:39.645200 26547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:39.645386 26547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "797bcd802d864865aa8b3d489ee7cae3"
format_stamp: "Formatted at 2026-09-25 15:03:39 on dist-test-slave-qxh0"
server_key: "a3bc93253bb45566727d8cf4d4feed0c"
server_key_iv: "2f53c16ba919cddc4cbec114cb69285f"
server_key_version: "encryptionkey@0"
I20260925 15:03:39.645596 26547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:39.678766 26547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:39.683174 26547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:39.683470 26547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:39.701751 26547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44855
I20260925 15:03:39.701764 26613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44855 every 8 connection(s)
I20260925 15:03:39.703217 26547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:39.706840 26614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:39.711705 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26547
I20260925 15:03:39.711908 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:39.712432 5071 external_mini_cluster.cc:1428] Setting key 8996b90f119e7f4c5857a6defed4c726
I20260925 15:03:39.717955 26614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3: Bootstrap starting.
I20260925 15:03:39.721457 26614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:39.722697 26614 log.cc:824] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:39.725625 26614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3: No bootstrap required, opened a new log
I20260925 15:03:39.725934 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:39.714164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:59341 (local address 127.0.0.1:44855)
0925 15:03:39.715013 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:39.715026 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:39.715066 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:03:39.716812 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:39.716834 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:39.717207 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:39.717504 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:39.718138 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:39.719503 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:39.721322 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:39.722108 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:39.722236 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:39.723607 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:39.723619 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:39.723640 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:39.724290 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:39.724372 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:39.724395 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:39.724505 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:39.724774 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:03:39.724986 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":428,"threads_started":1}
I20260925 15:03:39.729681 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:39.735077 26614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797bcd802d864865aa8b3d489ee7cae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260925 15:03:39.735437 26614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:39.735574 26614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797bcd802d864865aa8b3d489ee7cae3, State: Initialized, Role: FOLLOWER
I20260925 15:03:39.736141 26614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797bcd802d864865aa8b3d489ee7cae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260925 15:03:39.736423 26614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:39.736574 26614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:39.736788 26614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:39.739547 26614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797bcd802d864865aa8b3d489ee7cae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260925 15:03:39.740090 26614 leader_election.cc:302] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 797bcd802d864865aa8b3d489ee7cae3; no voters:
I20260925 15:03:39.740737 26614 leader_election.cc:288] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:39.740876 26624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:39.742455 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:39.734775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:44689 (local address 127.0.0.1:44855)
0925 15:03:39.734947 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:39.734952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:39.735392 (+ 440us) server_negotiation.cc:407] Connection header received
0925 15:03:39.735601 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:39.735606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:39.735685 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:39.735786 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:39.736344 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:39.737518 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:39.738927 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:39.739621 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:39.739749 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:39.741127 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:39.741135 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:39.741142 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:39.741180 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:39.741253 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:39.741285 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:39.741995 (+ 710us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:39.742071 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:03:39.742125 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:03:39.743109 26614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:39.747529 26624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [term 1 LEADER]: Becoming Leader. State: Replica: 797bcd802d864865aa8b3d489ee7cae3, State: Running, Role: LEADER
I20260925 15:03:39.748245 26624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797bcd802d864865aa8b3d489ee7cae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260925 15:03:39.750991 26625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "797bcd802d864865aa8b3d489ee7cae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797bcd802d864865aa8b3d489ee7cae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260925 15:03:39.751320 26625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:39.753018 26626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 797bcd802d864865aa8b3d489ee7cae3. Latest consensus state: current_term: 1 leader_uuid: "797bcd802d864865aa8b3d489ee7cae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "797bcd802d864865aa8b3d489ee7cae3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260925 15:03:39.753265 26626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3 [sys.catalog]: This master's current role is: LEADER
W20260925 15:03:39.756534 26638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:03:39.756630 26638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:03:39.756825 26639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:39.758375 26639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:39.766364 26639 catalog_manager.cc:1347] Generated new cluster ID: 731eb1d2282d4ae797648b5d11b803d7
I20260925 15:03:39.766470 26639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:39.781983 26639 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:39.783910 26639 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:39.794986 26639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 797bcd802d864865aa8b3d489ee7cae3: Generated new TSK 0
I20260925 15:03:39.795993 26639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:39.807403 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26547
W20260925 15:03:39.808504 26609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:59341
W20260925 15:03:39.809230 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:39.809397 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26547
W20260925 15:03:39.823562 26532 connection.cc:535] client connection to 127.0.0.1:44855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44855): Connection reset by peer (error 104)
2026-09-25T15:03:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:39 dist-test-slave-qxh0 krb5kdc[26645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:39 dist-test-slave-qxh0 krb5kdc[26645](info): set up 2 sockets
Sep 25 15:03:39 dist-test-slave-qxh0 krb5kdc[26645](info): commencing operation
krb5kdc: starting...
W20260925 15:03:40.915227 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:41 dist-test-slave-qxh0 krb5kdc[26645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:03:41.162492 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:41.486856 26658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:41.487141 26658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:41.487299 26658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:41.494381 26658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:41.494488 26658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:41.494565 26658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:41.494693 26658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:41.494765 26658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:41.504626 26658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:41.505996 26658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:41.507393 26658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:41.518548 26666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:41.518608 26664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:41.518833 26663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:41.519542 26658 server_base.cc:1034] running on GCE node
I20260925 15:03:41.520268 26658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:41.521543 26658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:41.522758 26658 hybrid_clock.cc:648] HybridClock initialized: now 1790348621522717 us; error 54 us; skew 500 ppm
I20260925 15:03:41.523096 26658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:41 dist-test-slave-qxh0 krb5kdc[26645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:41.533208 26658 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:03:41.534756 26658 webserver.cc:460] Webserver started at http://127.0.0.1:38107/ using document root <none> and password file <none>
I20260925 15:03:41.535422 26658 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:41.535565 26658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:41.535876 26658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:41.538820 26658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "05c32f9f59bf4e0592b5be1710b2a0b1"
format_stamp: "Formatted at 2026-09-25 15:03:41 on dist-test-slave-qxh0"
server_key: "c77958b3af5b3e12e8c06867ae61fa5a"
server_key_iv: "ebdb487fb371c428cae9a48b7ec160b2"
server_key_version: "encryptionkey@0"
I20260925 15:03:41.539562 26658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "05c32f9f59bf4e0592b5be1710b2a0b1"
format_stamp: "Formatted at 2026-09-25 15:03:41 on dist-test-slave-qxh0"
server_key: "c77958b3af5b3e12e8c06867ae61fa5a"
server_key_iv: "ebdb487fb371c428cae9a48b7ec160b2"
server_key_version: "encryptionkey@0"
I20260925 15:03:41.544665 26658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:41.548318 26673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:41.549657 26658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:03:41.549877 26658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "05c32f9f59bf4e0592b5be1710b2a0b1"
format_stamp: "Formatted at 2026-09-25 15:03:41 on dist-test-slave-qxh0"
server_key: "c77958b3af5b3e12e8c06867ae61fa5a"
server_key_iv: "ebdb487fb371c428cae9a48b7ec160b2"
server_key_version: "encryptionkey@0"
I20260925 15:03:41.550096 26658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:41.555689 26658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:41.559520 26658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:41.559815 26658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:41.577675 26658 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42129
I20260925 15:03:41.577723 26724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42129 every 8 connection(s)
I20260925 15:03:41.579064 26658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:41.582638 26725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:41.585639 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26658
I20260925 15:03:41.585840 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:41.586412 5071 external_mini_cluster.cc:1428] Setting key ed53729985711438c2ea424d844bd070
I20260925 15:03:41.593923 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:41.588294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:54797 (local address 127.0.0.1:42129)
0925 15:03:41.589031 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:41.589042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:41.589082 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:03:41.590722 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:41.590743 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:41.591068 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:41.591443 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:41.592439 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:41.592448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:41.592460 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:41.592478 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:41.592533 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:41.592552 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:41.592792 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:41.592934 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:03:41.593094 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":314,"threads_started":1}
I20260925 15:03:41.594578 26725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1: Bootstrap starting.
I20260925 15:03:41.597365 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:41.597755 26725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:41.598774 26725 log.cc:824] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:41.601258 26725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1: No bootstrap required, opened a new log
I20260925 15:03:41.604364 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:41.601730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.6:58701 (local address 127.0.0.1:42129)
0925 15:03:41.601955 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:41.601964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:41.602431 (+ 467us) server_negotiation.cc:407] Connection header received
0925 15:03:41.602584 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:41.602587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:41.602686 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:41.602766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:41.603679 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:41.603683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:41.603686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:41.603703 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:41.603737 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:41.603753 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:41.603865 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:41.604001 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:03:41.604054 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:03:41.609479 26725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260925 15:03:41.609838 26725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:41.609951 26725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c32f9f59bf4e0592b5be1710b2a0b1, State: Initialized, Role: FOLLOWER
I20260925 15:03:41.610548 26725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260925 15:03:41.610863 26725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:41.611011 26725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:41.611152 26725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:41.613255 26725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260925 15:03:41.613879 26725 leader_election.cc:302] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05c32f9f59bf4e0592b5be1710b2a0b1; no voters:
I20260925 15:03:41.614466 26725 leader_election.cc:288] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:41.614620 26735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:41.615273 26735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [term 1 LEADER]: Becoming Leader. State: Replica: 05c32f9f59bf4e0592b5be1710b2a0b1, State: Running, Role: LEADER
I20260925 15:03:41.615748 26735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } }
I20260925 15:03:41.616430 26725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:41.618646 26737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05c32f9f59bf4e0592b5be1710b2a0b1. Latest consensus state: current_term: 1 leader_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } } }
I20260925 15:03:41.618877 26736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05c32f9f59bf4e0592b5be1710b2a0b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42129 } } }
I20260925 15:03:41.619113 26737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:41.619195 26736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:41.620113 26743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:41.625594 26743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:41.634366 26743 catalog_manager.cc:1347] Generated new cluster ID: e902c4254c024fca983991653d52b0d5
I20260925 15:03:41.634486 26743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:41.653957 26743 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:41.654419 26743 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:41.664412 26743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05c32f9f59bf4e0592b5be1710b2a0b1: Generated new TSK 0
I20260925 15:03:41.665438 26743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:41.682438 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26658
W20260925 15:03:41.683593 26720 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.1.6:54797
W20260925 15:03:41.684269 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:41.684412 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26658
2026-09-25T15:03:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:41 dist-test-slave-qxh0 krb5kdc[26756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:41 dist-test-slave-qxh0 krb5kdc[26756](info): set up 2 sockets
Sep 25 15:03:41 dist-test-slave-qxh0 krb5kdc[26756](info): commencing operation
krb5kdc: starting...
W20260925 15:03:42.774999 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:42 dist-test-slave-qxh0 krb5kdc[26756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:43.023212 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35525
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:43.348217 26769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:43.348503 26769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:43.348667 26769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:43.355892 26769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:43.355983 26769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:43.356083 26769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:43.356194 26769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:43.356287 26769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:43.365967 26769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:43.367226 26769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:43.368480 26769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:43.380096 26777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:43.380144 26775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:43.380158 26774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:43.380833 26769 server_base.cc:1034] running on GCE node
I20260925 15:03:43.381544 26769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:43.382932 26769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:43.384135 26769 hybrid_clock.cc:648] HybridClock initialized: now 1790348623384074 us; error 51 us; skew 500 ppm
I20260925 15:03:43.384438 26769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:43 dist-test-slave-qxh0 krb5kdc[26756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348623, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:43.387660 26769 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:43.389178 26769 webserver.cc:460] Webserver started at http://127.4.243.254:43001/ using document root <none> and password file <none>
I20260925 15:03:43.389838 26769 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:43.389981 26769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:43.390321 26769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:43.393337 26769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "304791b9e4de40a2b6936fef08ea8bb1"
format_stamp: "Formatted at 2026-09-25 15:03:43 on dist-test-slave-qxh0"
server_key: "b3fb37d11d61f355f14f1fda15e45b02"
server_key_iv: "0865bd337d2e3ff8332a315d387a4951"
server_key_version: "encryptionkey@0"
I20260925 15:03:43.394112 26769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "304791b9e4de40a2b6936fef08ea8bb1"
format_stamp: "Formatted at 2026-09-25 15:03:43 on dist-test-slave-qxh0"
server_key: "b3fb37d11d61f355f14f1fda15e45b02"
server_key_iv: "0865bd337d2e3ff8332a315d387a4951"
server_key_version: "encryptionkey@0"
I20260925 15:03:43.399240 26769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:03:43.402705 26784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:43.403997 26769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:03:43.404181 26769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "304791b9e4de40a2b6936fef08ea8bb1"
format_stamp: "Formatted at 2026-09-25 15:03:43 on dist-test-slave-qxh0"
server_key: "b3fb37d11d61f355f14f1fda15e45b02"
server_key_iv: "0865bd337d2e3ff8332a315d387a4951"
server_key_version: "encryptionkey@0"
I20260925 15:03:43.404383 26769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:43.416563 26769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:43.420665 26769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:43.420954 26769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:43.439837 26769 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:35525
I20260925 15:03:43.439850 26835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:35525 every 8 connection(s)
I20260925 15:03:43.441214 26769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:43.444859 26836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:43.447080 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26769
I20260925 15:03:43.447257 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:43.447795 5071 external_mini_cluster.cc:1428] Setting key 99d11dfb374bd97fdb6535f03fce7128
I20260925 15:03:43.458698 26836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1: Bootstrap starting.
Sep 25 15:03:43 dist-test-slave-qxh0 krb5kdc[26756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:43.463248 26836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:43.464558 26836 log.cc:824] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:43.467576 26836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1: No bootstrap required, opened a new log
I20260925 15:03:43.471113 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:43.449566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33691 (local address 127.4.243.254:35525)
0925 15:03:43.450375 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:43.450386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:43.450432 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:03:43.452183 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:43.452204 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:43.452595 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:43.452898 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:43.454787 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:43.456115 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:43.457419 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:43.458209 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:43.458397 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:43.462831 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:43.462840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:43.462857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:43.462902 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:43.466181 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:43.467085 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:43.467092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:43.467099 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:43.467264 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:43.467796 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:43.467801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:43.467804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:43.468527 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:43.468858 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:43.469317 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 15:03:43.469680 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":401,"threads_started":1}
I20260925 15:03:43.475093 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:43.476763 26836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304791b9e4de40a2b6936fef08ea8bb1" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35525 } }
I20260925 15:03:43.477130 26836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:43.477226 26836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304791b9e4de40a2b6936fef08ea8bb1, State: Initialized, Role: FOLLOWER
I20260925 15:03:43.477872 26836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304791b9e4de40a2b6936fef08ea8bb1" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35525 } }
I20260925 15:03:43.478116 26836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:43.478274 26836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:43.478462 26836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:43.481220 26836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304791b9e4de40a2b6936fef08ea8bb1" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35525 } }
I20260925 15:03:43.481731 26836 leader_election.cc:302] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304791b9e4de40a2b6936fef08ea8bb1; no voters:
I20260925 15:03:43.482429 26836 leader_election.cc:288] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:43.482558 26846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:43.483254 26846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [term 1 LEADER]: Becoming Leader. State: Replica: 304791b9e4de40a2b6936fef08ea8bb1, State: Running, Role: LEADER
I20260925 15:03:43.483847 26846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304791b9e4de40a2b6936fef08ea8bb1" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35525 } }
I20260925 15:03:43.485554 26846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304791b9e4de40a2b6936fef08ea8bb1. Latest consensus state: current_term: 1 leader_uuid: "304791b9e4de40a2b6936fef08ea8bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304791b9e4de40a2b6936fef08ea8bb1" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35525 } } }
I20260925 15:03:43.485829 26846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:43.486438 26848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "304791b9e4de40a2b6936fef08ea8bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304791b9e4de40a2b6936fef08ea8bb1" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35525 } } }
I20260925 15:03:43.486845 26848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:43.501291 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:43.479500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58221 (local address 127.4.243.254:35525)
0925 15:03:43.479692 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:43.479698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:43.480092 (+ 394us) server_negotiation.cc:407] Connection header received
0925 15:03:43.480228 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:43.480233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:43.480327 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:43.480434 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:43.481871 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:43.483252 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:43.485714 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:43.486710 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:43.490731 (+ 4021us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:43.491085 (+ 354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:43.491094 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:43.491102 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:43.491142 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:43.495873 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:43.497274 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:43.497282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:43.497288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:43.497504 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:43.498194 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:43.498201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:43.498206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:43.498722 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:43.500265 (+ 1543us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:43.500495 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:03:43.500755 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:03:43.503237 26836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:43.503502 26850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:43.508838 26850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:43.518134 26850 catalog_manager.cc:1347] Generated new cluster ID: 8200834d7dfd4d23abd43b9aba7496a3
I20260925 15:03:43.518246 26850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:43.539057 26850 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:43.540306 26850 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:43.556277 26850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 304791b9e4de40a2b6936fef08ea8bb1: Generated new TSK 0
I20260925 15:03:43.557324 26850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:43.564663 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26769
I20260925 15:03:43.566108 26831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:43.657787 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26769
W20260925 15:03:43.675160 26754 connection.cc:535] client connection to 127.4.243.254:35525 recv error: Network error: failed to read from TLS socket (remote: 127.4.243.254:35525): Connection reset by peer (error 104)
2026-09-25T15:03:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:43 dist-test-slave-qxh0 krb5kdc[26868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:43 dist-test-slave-qxh0 krb5kdc[26868](info): set up 2 sockets
Sep 25 15:03:43 dist-test-slave-qxh0 krb5kdc[26868](info): commencing operation
krb5kdc: starting...
W20260925 15:03:44.742959 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:44 dist-test-slave-qxh0 krb5kdc[26868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:44.982336 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:42797
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:45.297801 26881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:45.298062 26881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:45.298147 26881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:45.304970 26881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:45.305081 26881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:45.305126 26881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:45.305186 26881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:45.305228 26881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:45.314829 26881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:42797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:45.315990 26881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:45.317178 26881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:45.328541 26886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:45.328578 26889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:45.328647 26887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:45.329357 26881 server_base.cc:1034] running on GCE node
I20260925 15:03:45.330021 26881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:45.331254 26881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:45.332453 26881 hybrid_clock.cc:648] HybridClock initialized: now 1790348625332415 us; error 27 us; skew 500 ppm
I20260925 15:03:45.332783 26881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:45 dist-test-slave-qxh0 krb5kdc[26868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348625, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:45.335845 26881 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:45.337146 26881 webserver.cc:460] Webserver started at http://127.4.243.254:41771/ using document root <none> and password file <none>
I20260925 15:03:45.337749 26881 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:45.337875 26881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:45.338148 26881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:45.340878 26881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "da586b7c82294ca08d9d8e5b85e22afe"
format_stamp: "Formatted at 2026-09-25 15:03:45 on dist-test-slave-qxh0"
server_key: "bdd47c712f6c90b321091e263da2113d"
server_key_iv: "5a22b8c60dd1529fd257d4736b0bd91f"
server_key_version: "encryptionkey@0"
I20260925 15:03:45.341573 26881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "da586b7c82294ca08d9d8e5b85e22afe"
format_stamp: "Formatted at 2026-09-25 15:03:45 on dist-test-slave-qxh0"
server_key: "bdd47c712f6c90b321091e263da2113d"
server_key_iv: "5a22b8c60dd1529fd257d4736b0bd91f"
server_key_version: "encryptionkey@0"
I20260925 15:03:45.346730 26881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 15:03:45.350064 26896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:45.351423 26881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:45.351608 26881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "da586b7c82294ca08d9d8e5b85e22afe"
format_stamp: "Formatted at 2026-09-25 15:03:45 on dist-test-slave-qxh0"
server_key: "bdd47c712f6c90b321091e263da2113d"
server_key_iv: "5a22b8c60dd1529fd257d4736b0bd91f"
server_key_version: "encryptionkey@0"
I20260925 15:03:45.351801 26881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:45.376422 26881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:45.380334 26881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:45.380612 26881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:45.399264 26881 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:42797
I20260925 15:03:45.399282 26947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:42797 every 8 connection(s)
I20260925 15:03:45.400640 26881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:45.404304 26948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:45.404501 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26881
I20260925 15:03:45.404681 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:45.405098 5071 external_mini_cluster.cc:1428] Setting key 97fe565b0546ba990b23340c17883b17
I20260925 15:03:45.419513 26948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe: Bootstrap starting.
I20260925 15:03:45.419708 26951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:45.406916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36549 (local address 127.4.243.254:42797)
0925 15:03:45.407727 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:45.407743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:45.407796 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:03:45.409604 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:45.409635 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:45.410185 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:45.410661 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:45.411181 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:45.413256 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:45.414559 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:45.415678 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:45.415873 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:45.416947 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:45.416957 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:45.416975 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:45.417690 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:45.417780 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:45.417809 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:45.417981 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:45.418273 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:03:45.418436 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260925 15:03:45.422895 26948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:45.423679 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:45.424108 26948 log.cc:824] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:45.427239 26948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe: No bootstrap required, opened a new log
I20260925 15:03:45.434811 26951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:45.428564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51883 (local address 127.4.243.254:42797)
0925 15:03:45.428766 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:45.428773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:45.429084 (+ 311us) server_negotiation.cc:407] Connection header received
0925 15:03:45.429349 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:45.429353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:45.429414 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:45.429528 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:45.430064 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:45.430925 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:45.432235 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:45.432718 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:45.432868 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:45.434004 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:45.434010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:45.434014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:45.434030 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:45.434069 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:45.434085 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:45.434219 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:45.434416 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 15:03:45.434469 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:03:45.435201 26948 raft_consensus.cc:348] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da586b7c82294ca08d9d8e5b85e22afe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 42797 } }
I20260925 15:03:45.435586 26948 raft_consensus.cc:374] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:45.435704 26948 raft_consensus.cc:729] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da586b7c82294ca08d9d8e5b85e22afe, State: Initialized, Role: FOLLOWER
I20260925 15:03:45.436367 26948 consensus_queue.cc:260] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da586b7c82294ca08d9d8e5b85e22afe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 42797 } }
I20260925 15:03:45.436631 26948 raft_consensus.cc:388] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:45.436782 26948 raft_consensus.cc:482] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:45.436964 26948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:45.439206 26948 raft_consensus.cc:504] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da586b7c82294ca08d9d8e5b85e22afe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 42797 } }
I20260925 15:03:45.439692 26948 leader_election.cc:302] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da586b7c82294ca08d9d8e5b85e22afe; no voters:
I20260925 15:03:45.440254 26948 leader_election.cc:288] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:45.440394 26958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:45.441149 26958 raft_consensus.cc:686] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [term 1 LEADER]: Becoming Leader. State: Replica: da586b7c82294ca08d9d8e5b85e22afe, State: Running, Role: LEADER
I20260925 15:03:45.441669 26958 consensus_queue.cc:237] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da586b7c82294ca08d9d8e5b85e22afe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 42797 } }
I20260925 15:03:45.442747 26948 sys_catalog.cc:564] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:45.444770 26960 sys_catalog.cc:455] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [sys.catalog]: SysCatalogTable state changed. Reason: New leader da586b7c82294ca08d9d8e5b85e22afe. Latest consensus state: current_term: 1 leader_uuid: "da586b7c82294ca08d9d8e5b85e22afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da586b7c82294ca08d9d8e5b85e22afe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 42797 } } }
I20260925 15:03:45.444744 26959 sys_catalog.cc:455] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da586b7c82294ca08d9d8e5b85e22afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da586b7c82294ca08d9d8e5b85e22afe" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 42797 } } }
I20260925 15:03:45.445190 26960 sys_catalog.cc:458] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:45.445190 26959 sys_catalog.cc:458] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:45.446046 26964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:45.450398 26964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:45.460206 26964 catalog_manager.cc:1347] Generated new cluster ID: c5581f4ca3d444cb860e3d7fad21b97a
I20260925 15:03:45.460333 26964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:45.481899 26964 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:45.483309 26964 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:45.490643 26964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da586b7c82294ca08d9d8e5b85e22afe: Generated new TSK 0
I20260925 15:03:45.491433 26964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:45.503721 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26881
W20260925 15:03:45.504895 26943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36549
W20260925 15:03:45.505716 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26881: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:45.505879 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26881
W20260925 15:03:45.523454 26866 connection.cc:535] client connection to 127.4.243.254:42797 recv error: Network error: failed to read from TLS socket (remote: 127.4.243.254:42797): Connection reset by peer (error 104)
2026-09-25T15:03:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:45 dist-test-slave-qxh0 krb5kdc[26979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:45 dist-test-slave-qxh0 krb5kdc[26979](info): set up 2 sockets
Sep 25 15:03:45 dist-test-slave-qxh0 krb5kdc[26979](info): commencing operation
krb5kdc: starting...
W20260925 15:03:46.594988 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:46 dist-test-slave-qxh0 krb5kdc[26979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:46.825217 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:45877
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:47.142067 26992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:47.142329 26992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:47.142508 26992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:47.149641 26992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:47.149745 26992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:47.149873 26992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:47.149981 26992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:47.150074 26992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:47.159420 26992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:45877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:47.160594 26992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:47.161815 26992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:47.172982 26997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:47.172955 26998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:47.173198 27000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:47.173977 26992 server_base.cc:1034] running on GCE node
I20260925 15:03:47.174762 26992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:47.176036 26992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:47.177320 26992 hybrid_clock.cc:648] HybridClock initialized: now 1790348627177271 us; error 31 us; skew 500 ppm
I20260925 15:03:47.177707 26992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:47 dist-test-slave-qxh0 krb5kdc[26979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348627, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:47.184862 26992 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:47.186534 26992 webserver.cc:460] Webserver started at http://127.4.243.254:41005/ using document root <none> and password file <none>
I20260925 15:03:47.187268 26992 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:47.187425 26992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:47.187732 26992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:47.190750 26992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59"
format_stamp: "Formatted at 2026-09-25 15:03:47 on dist-test-slave-qxh0"
server_key: "2d92d04210138568dde2699f84d42fc9"
server_key_iv: "3370707f2c79c0d7109b09dbdaceed81"
server_key_version: "encryptionkey@0"
I20260925 15:03:47.191689 26992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59"
format_stamp: "Formatted at 2026-09-25 15:03:47 on dist-test-slave-qxh0"
server_key: "2d92d04210138568dde2699f84d42fc9"
server_key_iv: "3370707f2c79c0d7109b09dbdaceed81"
server_key_version: "encryptionkey@0"
I20260925 15:03:47.196740 26992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:47.199891 27007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:47.201054 26992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:03:47.201220 26992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59"
format_stamp: "Formatted at 2026-09-25 15:03:47 on dist-test-slave-qxh0"
server_key: "2d92d04210138568dde2699f84d42fc9"
server_key_iv: "3370707f2c79c0d7109b09dbdaceed81"
server_key_version: "encryptionkey@0"
I20260925 15:03:47.201406 26992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:47.206715 26992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:47.210803 26992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:47.211076 26992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:47.229171 26992 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:45877
I20260925 15:03:47.229211 27058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:45877 every 8 connection(s)
I20260925 15:03:47.230518 26992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:47.234074 27059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:47.236495 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 26992
I20260925 15:03:47.236650 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:47.237185 5071 external_mini_cluster.cc:1428] Setting key 07b8fa683a39af42f7c843b5aefe05e3
I20260925 15:03:47.244609 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:47.238961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56717 (local address 127.4.243.254:45877)
0925 15:03:47.239705 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:47.239718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:47.239757 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:03:47.241283 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:47.241303 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:47.241610 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:47.241936 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:47.243063 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:47.243069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:47.243084 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:47.243115 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:47.243175 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:47.243194 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:47.243382 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:47.243483 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:03:47.243711 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":371,"threads_started":1}
I20260925 15:03:47.246003 27059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59: Bootstrap starting.
I20260925 15:03:47.248142 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:47.249431 27059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:47.250698 27059 log.cc:824] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:47.253445 27059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59: No bootstrap required, opened a new log
I20260925 15:03:47.255687 27062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:47.253342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34271 (local address 127.4.243.254:45877)
0925 15:03:47.253506 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:47.253511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:47.253891 (+ 380us) server_negotiation.cc:407] Connection header received
0925 15:03:47.254013 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:47.254018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:47.254080 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:47.254189 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:47.255065 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:47.255070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:47.255074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:47.255091 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:47.255147 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:47.255174 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:47.255290 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:47.255348 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:03:47.255398 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:03:47.261581 27059 raft_consensus.cc:348] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45877 } }
I20260925 15:03:47.261960 27059 raft_consensus.cc:374] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:47.262082 27059 raft_consensus.cc:729] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36a15c8e9f84fc5a0f5d49c90e45e59, State: Initialized, Role: FOLLOWER
I20260925 15:03:47.262841 27059 consensus_queue.cc:260] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45877 } }
I20260925 15:03:47.263079 27069 raft_consensus.cc:482] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:03:47.263325 27069 raft_consensus.cc:504] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45877 } }
I20260925 15:03:47.263849 27059 raft_consensus.cc:388] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:47.263983 27059 raft_consensus.cc:482] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:47.263887 27069 leader_election.cc:302] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c36a15c8e9f84fc5a0f5d49c90e45e59; no voters:
I20260925 15:03:47.264127 27059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:47.264720 27069 leader_election.cc:288] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260925 15:03:47.266435 27059 raft_consensus.cc:504] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45877 } }
I20260925 15:03:47.266778 27059 leader_election.cc:302] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c36a15c8e9f84fc5a0f5d49c90e45e59; no voters:
I20260925 15:03:47.266844 27070 raft_consensus.cc:2741] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 15:03:47.267031 27059 leader_election.cc:288] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:47.267369 27069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:47.268652 27069 raft_consensus.cc:686] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [term 1 LEADER]: Becoming Leader. State: Replica: c36a15c8e9f84fc5a0f5d49c90e45e59, State: Running, Role: LEADER
I20260925 15:03:47.269091 27069 consensus_queue.cc:237] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45877 } }
I20260925 15:03:47.269512 27059 sys_catalog.cc:564] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:47.271107 27070 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c36a15c8e9f84fc5a0f5d49c90e45e59. Latest consensus state: current_term: 1 leader_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45877 } } }
I20260925 15:03:47.271600 27070 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:47.272043 27071 sys_catalog.cc:455] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c36a15c8e9f84fc5a0f5d49c90e45e59" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 45877 } } }
I20260925 15:03:47.272399 27071 sys_catalog.cc:458] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:47.273540 27077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:47.277961 27077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:47.285962 27077 catalog_manager.cc:1347] Generated new cluster ID: 1618b226dd164434b7980c4e5dc85966
I20260925 15:03:47.286084 27077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:47.324836 27077 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:47.325335 27077 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:47.340229 27077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c36a15c8e9f84fc5a0f5d49c90e45e59: Generated new TSK 0
I20260925 15:03:47.341084 27077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:47.354056 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26992
W20260925 15:03:47.355194 27054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56717
W20260925 15:03:47.355894 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 26992: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:47.356037 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 26992
2026-09-25T15:03:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:47 dist-test-slave-qxh0 krb5kdc[27091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:47 dist-test-slave-qxh0 krb5kdc[27091](info): set up 2 sockets
Sep 25 15:03:47 dist-test-slave-qxh0 krb5kdc[27091](info): commencing operation
krb5kdc: starting...
W20260925 15:03:48.455153 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:48 dist-test-slave-qxh0 krb5kdc[27091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:48.696305 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:32863
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:36367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:32863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:49.020088 27104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:49.020359 27104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:49.020572 27104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:49.027841 27104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:49.027961 27104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:49.028085 27104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:49.028196 27104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:49.028283 27104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:49.037899 27104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:36367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:32863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:49.039225 27104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:49.040527 27104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:49.052150 27109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:49.052155 27110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:49.052628 27112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:49.053179 27104 server_base.cc:1034] running on GCE node
I20260925 15:03:49.053961 27104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:49.055347 27104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:49.056572 27104 hybrid_clock.cc:648] HybridClock initialized: now 1790348629056520 us; error 37 us; skew 500 ppm
I20260925 15:03:49.056928 27104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:49 dist-test-slave-qxh0 krb5kdc[27091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348629, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:49.059949 27104 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:49.061331 27104 webserver.cc:460] Webserver started at http://127.4.243.254:36011/ using document root <none> and password file <none>
I20260925 15:03:49.061977 27104 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:49.062094 27104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:49.062335 27104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:49.065277 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e"
format_stamp: "Formatted at 2026-09-25 15:03:49 on dist-test-slave-qxh0"
server_key: "69d7616b428413cb908124e2313cd587"
server_key_iv: "ebd73eb2aa6406a23801d14b7c35f7f8"
server_key_version: "encryptionkey@0"
I20260925 15:03:49.066049 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e"
format_stamp: "Formatted at 2026-09-25 15:03:49 on dist-test-slave-qxh0"
server_key: "69d7616b428413cb908124e2313cd587"
server_key_iv: "ebd73eb2aa6406a23801d14b7c35f7f8"
server_key_version: "encryptionkey@0"
I20260925 15:03:49.071179 27104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 15:03:49.074414 27119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:49.075739 27104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:49.075887 27104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e"
format_stamp: "Formatted at 2026-09-25 15:03:49 on dist-test-slave-qxh0"
server_key: "69d7616b428413cb908124e2313cd587"
server_key_iv: "ebd73eb2aa6406a23801d14b7c35f7f8"
server_key_version: "encryptionkey@0"
I20260925 15:03:49.076056 27104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:49.103942 27104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:49.107973 27104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:49.108210 27104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:49.126195 27104 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:32863
I20260925 15:03:49.126219 27170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:32863 every 8 connection(s)
I20260925 15:03:49.127516 27104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:49.128882 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27104
I20260925 15:03:49.129079 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:49.129626 5071 external_mini_cluster.cc:1428] Setting key 43fd4b4168ae39e1baab0ec81b16ffad
I20260925 15:03:49.131994 27171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:03:49 dist-test-slave-qxh0 krb5kdc[27091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:49.144541 27171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e: Bootstrap starting.
I20260925 15:03:49.148696 27171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:49.149889 27171 log.cc:824] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:49.153151 27171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e: No bootstrap required, opened a new log
I20260925 15:03:49.153275 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:49.131412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58381 (local address 127.4.243.254:32863)
0925 15:03:49.132241 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:49.132251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:49.132292 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:03:49.133812 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:49.133832 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:49.134210 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:49.134537 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:49.136244 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:49.137557 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:49.139284 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:49.139886 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:49.140005 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:49.144607 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:49.144620 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:49.144639 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:49.144682 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:49.148684 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:49.149451 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:49.149459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:49.149467 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:49.149684 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:49.150150 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:49.150155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:49.150158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:49.150878 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:49.151108 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:49.151648 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 15:03:49.152091 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":340,"threads_started":1}
I20260925 15:03:49.156826 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:49.160845 27171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 32863 } }
I20260925 15:03:49.161192 27171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:49.161293 27171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54aa1416f4a343a8a7b3d2cb5b260e6e, State: Initialized, Role: FOLLOWER
I20260925 15:03:49.161888 27171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 32863 } }
I20260925 15:03:49.162122 27171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:49.162252 27171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:49.162408 27171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:49.165042 27171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 32863 } }
I20260925 15:03:49.165499 27171 leader_election.cc:302] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54aa1416f4a343a8a7b3d2cb5b260e6e; no voters:
I20260925 15:03:49.166121 27171 leader_election.cc:288] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:49.166242 27181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:49.166905 27181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [term 1 LEADER]: Becoming Leader. State: Replica: 54aa1416f4a343a8a7b3d2cb5b260e6e, State: Running, Role: LEADER
I20260925 15:03:49.167506 27181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 32863 } }
I20260925 15:03:49.170125 27183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 32863 } } }
I20260925 15:03:49.170537 27183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:49.172739 27183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54aa1416f4a343a8a7b3d2cb5b260e6e. Latest consensus state: current_term: 1 leader_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa1416f4a343a8a7b3d2cb5b260e6e" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 32863 } } }
I20260925 15:03:49.172996 27183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:49.174765 27171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:49.176465 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:49.160740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42303 (local address 127.4.243.254:32863)
0925 15:03:49.160931 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:49.160936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:49.161427 (+ 491us) server_negotiation.cc:407] Connection header received
0925 15:03:49.161573 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:49.161578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:49.161645 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:49.161759 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:49.163185 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:49.164302 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:49.165604 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:49.166332 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:49.166478 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:49.169456 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:49.169463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:49.169468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:49.169511 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:49.173502 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:49.174280 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:49.174287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:49.174291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:49.174469 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:49.174935 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:49.174940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:49.174943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:49.175276 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:49.175596 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:49.175940 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:03:49.176139 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:03:49.176620 27185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:49.182438 27185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:49.191602 27185 catalog_manager.cc:1347] Generated new cluster ID: 10cff2d60f8042a4a717566fa41ff99c
I20260925 15:03:49.191723 27185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:49.204662 27185 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:49.206333 27185 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:49.213140 27185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54aa1416f4a343a8a7b3d2cb5b260e6e: Generated new TSK 0
I20260925 15:03:49.214090 27185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:49.227656 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27104
I20260925 15:03:49.228911 27166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:49.317122 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27104
2026-09-25T15:03:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:49 dist-test-slave-qxh0 krb5kdc[27204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:49 dist-test-slave-qxh0 krb5kdc[27204](info): set up 2 sockets
Sep 25 15:03:49 dist-test-slave-qxh0 krb5kdc[27204](info): commencing operation
krb5kdc: starting...
W20260925 15:03:50.415096 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:50 dist-test-slave-qxh0 krb5kdc[27204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:50.661552 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:34595
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:50.974803 27217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:50.975131 27217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:50.975299 27217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:50.982147 27217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:50.982281 27217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:50.982394 27217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:50.982493 27217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:50.982577 27217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:50.992162 27217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:34595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:34595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:50.993468 27217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:50.994858 27217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:51.006153 27225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:51.006197 27223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:51.006201 27222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:51.007057 27217 server_base.cc:1034] running on GCE node
I20260925 15:03:51.007769 27217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:51.009049 27217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:51.010242 27217 hybrid_clock.cc:648] HybridClock initialized: now 1790348631010195 us; error 39 us; skew 500 ppm
I20260925 15:03:51.010581 27217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:51 dist-test-slave-qxh0 krb5kdc[27204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348631, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:51.013651 27217 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:51.015086 27217 webserver.cc:460] Webserver started at http://127.4.243.254:46685/ using document root <none> and password file <none>
I20260925 15:03:51.015722 27217 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:51.015844 27217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:51.016178 27217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:51.019088 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "8de72365c0bc49aa9dfbbc83ac74afbf"
format_stamp: "Formatted at 2026-09-25 15:03:51 on dist-test-slave-qxh0"
server_key: "13468e06b22f42eb4c7c4d1be324c818"
server_key_iv: "c08dca7f570a68493a8d023532e31235"
server_key_version: "encryptionkey@0"
I20260925 15:03:51.019807 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "8de72365c0bc49aa9dfbbc83ac74afbf"
format_stamp: "Formatted at 2026-09-25 15:03:51 on dist-test-slave-qxh0"
server_key: "13468e06b22f42eb4c7c4d1be324c818"
server_key_iv: "c08dca7f570a68493a8d023532e31235"
server_key_version: "encryptionkey@0"
I20260925 15:03:51.024569 27217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:03:51.027906 27232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:51.029209 27217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:03:51.029385 27217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "8de72365c0bc49aa9dfbbc83ac74afbf"
format_stamp: "Formatted at 2026-09-25 15:03:51 on dist-test-slave-qxh0"
server_key: "13468e06b22f42eb4c7c4d1be324c818"
server_key_iv: "c08dca7f570a68493a8d023532e31235"
server_key_version: "encryptionkey@0"
I20260925 15:03:51.029580 27217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:51.054421 27217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:51.058841 27217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:51.059074 27217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:51.077447 27217 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:34595
I20260925 15:03:51.077471 27283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:34595 every 8 connection(s)
I20260925 15:03:51.078871 27217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:51.082350 27284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:51.083599 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27217
I20260925 15:03:51.083760 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:51.084161 5071 external_mini_cluster.cc:1428] Setting key 396ca42c980568c166566731c90ee232
I20260925 15:03:51.095440 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf: Bootstrap starting.
I20260925 15:03:51.097203 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:51.086028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35149 (local address 127.4.243.254:34595)
0925 15:03:51.086811 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:51.086824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:51.086862 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:03:51.088340 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:51.088362 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:51.088725 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:51.089055 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:51.089623 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:51.090907 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:51.092865 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:51.093560 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:51.093698 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:51.095057 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:51.095064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:51.095077 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:51.095636 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:51.095716 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:51.095740 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:51.095830 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:51.096053 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 15:03:51.096301 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":358,"threads_started":1}
I20260925 15:03:51.098846 27284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:51.099973 27284 log.cc:824] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:51.100852 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:51.102525 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf: No bootstrap required, opened a new log
I20260925 15:03:51.110445 27284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34595 } }
I20260925 15:03:51.110908 27284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:51.111050 27284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de72365c0bc49aa9dfbbc83ac74afbf, State: Initialized, Role: FOLLOWER
I20260925 15:03:51.111238 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:51.105007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53653 (local address 127.4.243.254:34595)
0925 15:03:51.105190 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:51.105197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:51.105534 (+ 337us) server_negotiation.cc:407] Connection header received
0925 15:03:51.105714 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:51.105720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:51.105782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:51.105889 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:51.106424 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:51.107239 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:51.108564 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:51.109040 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:51.109198 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:51.110407 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:51.110412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:51.110415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:51.110435 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:51.110476 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:51.110491 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:51.110587 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:51.110751 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:03:51.110834 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:03:51.111726 27284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34595 } }
I20260925 15:03:51.111984 27284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:51.112143 27284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:51.112313 27284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:51.114670 27284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34595 } }
I20260925 15:03:51.115244 27284 leader_election.cc:302] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8de72365c0bc49aa9dfbbc83ac74afbf; no voters:
I20260925 15:03:51.115890 27284 leader_election.cc:288] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:51.115994 27294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:51.116657 27294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [term 1 LEADER]: Becoming Leader. State: Replica: 8de72365c0bc49aa9dfbbc83ac74afbf, State: Running, Role: LEADER
I20260925 15:03:51.117238 27294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34595 } }
I20260925 15:03:51.118124 27284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:51.120172 27296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8de72365c0bc49aa9dfbbc83ac74afbf. Latest consensus state: current_term: 1 leader_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34595 } } }
I20260925 15:03:51.121506 27296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:51.121043 27295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de72365c0bc49aa9dfbbc83ac74afbf" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 34595 } } }
I20260925 15:03:51.122673 27295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:51.123167 27301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:51.127918 27301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:51.136094 27301 catalog_manager.cc:1347] Generated new cluster ID: e9475cb5e463425cb2153e3a84ab8763
I20260925 15:03:51.136183 27301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:51.147363 27301 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:51.148630 27301 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:51.155233 27301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8de72365c0bc49aa9dfbbc83ac74afbf: Generated new TSK 0
I20260925 15:03:51.156008 27301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:51.163537 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27217
W20260925 15:03:51.164673 27279 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35149
W20260925 15:03:51.165318 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:51.165462 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27217
2026-09-25T15:03:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:51 dist-test-slave-qxh0 krb5kdc[27315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:51 dist-test-slave-qxh0 krb5kdc[27315](info): set up 2 sockets
Sep 25 15:03:51 dist-test-slave-qxh0 krb5kdc[27315](info): commencing operation
krb5kdc: starting...
W20260925 15:03:52.239086 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:52 dist-test-slave-qxh0 krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:52.474730 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:46199
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:46199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:52.795601 27328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:52.795871 27328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:52.795957 27328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:52.802907 27328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:52.803011 27328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:52.803062 27328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:52.803120 27328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:52.803164 27328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:52.812435 27328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:46199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:52.813567 27328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:52.814785 27328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:52.825860 27334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:52.825968 27336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:52.825937 27333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:52.831979 27328 server_base.cc:1034] running on GCE node
I20260925 15:03:52.832808 27328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:52.834247 27328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:52.835482 27328 hybrid_clock.cc:648] HybridClock initialized: now 1790348632835439 us; error 51 us; skew 500 ppm
I20260925 15:03:52.835820 27328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:52 dist-test-slave-qxh0 krb5kdc[27315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348632, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:52.843252 27328 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:52.844749 27328 webserver.cc:460] Webserver started at http://127.4.243.254:37123/ using document root <none> and password file <none>
I20260925 15:03:52.845420 27328 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:52.845559 27328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:52.845851 27328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:52.848819 27328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "1a1cf7fdc1524ce098673eff8ad8bba7"
format_stamp: "Formatted at 2026-09-25 15:03:52 on dist-test-slave-qxh0"
server_key: "f21148d79b1cd24acc82d79cf5b87564"
server_key_iv: "8c310e4e764bfd55f60cd3c1130d7f95"
server_key_version: "encryptionkey@0"
I20260925 15:03:52.849547 27328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "1a1cf7fdc1524ce098673eff8ad8bba7"
format_stamp: "Formatted at 2026-09-25 15:03:52 on dist-test-slave-qxh0"
server_key: "f21148d79b1cd24acc82d79cf5b87564"
server_key_iv: "8c310e4e764bfd55f60cd3c1130d7f95"
server_key_version: "encryptionkey@0"
I20260925 15:03:52.854447 27328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 15:03:52.857705 27343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:52.858973 27328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:52.859159 27328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "1a1cf7fdc1524ce098673eff8ad8bba7"
format_stamp: "Formatted at 2026-09-25 15:03:52 on dist-test-slave-qxh0"
server_key: "f21148d79b1cd24acc82d79cf5b87564"
server_key_iv: "8c310e4e764bfd55f60cd3c1130d7f95"
server_key_version: "encryptionkey@0"
I20260925 15:03:52.859345 27328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:52.864790 27328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:52.868553 27328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:52.868836 27328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:52.886829 27328 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:46199
I20260925 15:03:52.886830 27394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:46199 every 8 connection(s)
I20260925 15:03:52.888253 27328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:52.891788 27395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:52.897254 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27328
I20260925 15:03:52.897419 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:52.897851 5071 external_mini_cluster.cc:1428] Setting key d83b62fdb136f860e6a8fdb6df925f4e
I20260925 15:03:52.901809 27395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7: Bootstrap starting.
I20260925 15:03:52.905151 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:52.899687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40433 (local address 127.4.243.254:46199)
0925 15:03:52.900481 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:52.900497 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:52.900533 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:03:52.902107 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:52.902128 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:52.902431 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:52.902803 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:52.903754 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:52.903760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:52.903771 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:52.903783 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:52.903839 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:52.903860 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:52.904014 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:52.904104 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 15:03:52.904291 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":405,"threads_started":1}
I20260925 15:03:52.905237 27395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:52.906260 27395 log.cc:824] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:52.908725 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:52.909209 27395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7: No bootstrap required, opened a new log
I20260925 15:03:52.915388 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:52.912911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60763 (local address 127.4.243.254:46199)
0925 15:03:52.913080 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:52.913089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:52.913445 (+ 356us) server_negotiation.cc:407] Connection header received
0925 15:03:52.913624 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:52.913628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:52.913674 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:52.913773 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:52.914721 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:52.914726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:52.914731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:03:52.914752 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:52.914798 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:03:52.914813 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:52.914960 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:52.915049 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:03:52.915108 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:03:52.917855 27395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 46199 } }
I20260925 15:03:52.918275 27395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:03:52.918371 27395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1cf7fdc1524ce098673eff8ad8bba7, State: Initialized, Role: FOLLOWER
I20260925 15:03:52.919065 27395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 46199 } }
I20260925 15:03:52.919318 27395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:03:52.919414 27395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:03:52.919538 27395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:52.921546 27395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 46199 } }
I20260925 15:03:52.921965 27395 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a1cf7fdc1524ce098673eff8ad8bba7; no voters:
I20260925 15:03:52.922480 27395 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:03:52.922642 27405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:52.923609 27405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [term 1 LEADER]: Becoming Leader. State: Replica: 1a1cf7fdc1524ce098673eff8ad8bba7, State: Running, Role: LEADER
I20260925 15:03:52.924165 27405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 46199 } }
I20260925 15:03:52.924891 27395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:52.927161 27406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 46199 } } }
I20260925 15:03:52.927477 27406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:52.928210 27407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a1cf7fdc1524ce098673eff8ad8bba7. Latest consensus state: current_term: 1 leader_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1cf7fdc1524ce098673eff8ad8bba7" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 46199 } } }
I20260925 15:03:52.928629 27407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:52.928959 27412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:52.935041 27412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:52.945111 27412 catalog_manager.cc:1347] Generated new cluster ID: e9510ba725cc4cc2afcada5ad475b307
I20260925 15:03:52.945214 27412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:53.012516 27412 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:53.012943 27412 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:53.022630 27412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a1cf7fdc1524ce098673eff8ad8bba7: Generated new TSK 0
I20260925 15:03:53.023366 27412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:53.039031 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27328
W20260925 15:03:53.040110 27390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40433
W20260925 15:03:53.040783 5071 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:03:53.040930 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27328
2026-09-25T15:03:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1878 ms)
[----------] 18 tests from AuthTokenIssuingTest (34059 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:53 dist-test-slave-qxh0 krb5kdc[27426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:53 dist-test-slave-qxh0 krb5kdc[27426](info): set up 2 sockets
Sep 25 15:03:53 dist-test-slave-qxh0 krb5kdc[27426](info): commencing operation
krb5kdc: starting...
W20260925 15:03:54.111192 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:54 dist-test-slave-qxh0 krb5kdc[27426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:54.357394 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:41435
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:41435,127.4.243.253:45939,127.4.243.252:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:54.676503 27439 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:03:54.676783 27439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:54.676847 27439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:54.676920 27439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:54.683585 27439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:54.683657 27439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:54.683708 27439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:54.683753 27439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:54.683805 27439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:54.692865 27439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:41435,127.4.243.253:45939,127.4.243.252:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:41435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:54.693998 27439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:54.695228 27439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:54.706821 27444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:54.706915 27445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:54.706871 27447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:54.707650 27439 server_base.cc:1034] running on GCE node
I20260925 15:03:54.708388 27439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:54.709802 27439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:54.711046 27439 hybrid_clock.cc:648] HybridClock initialized: now 1790348634710966 us; error 54 us; skew 500 ppm
I20260925 15:03:54.711390 27439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:54 dist-test-slave-qxh0 krb5kdc[27426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:54.714373 27439 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:54.715855 27439 webserver.cc:460] Webserver started at http://127.4.243.254:35285/ using document root <none> and password file <none>
I20260925 15:03:54.716526 27439 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:54.716660 27439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:54.716957 27439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:54.719924 27439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "896b18a03ab04ea6a056c9a62b3995ca"
format_stamp: "Formatted at 2026-09-25 15:03:54 on dist-test-slave-qxh0"
server_key: "7a0e76892633f824b447d97fe6905255"
server_key_iv: "0d461338e940b37bdbb2061ec779dd1f"
server_key_version: "encryptionkey@0"
I20260925 15:03:54.720698 27439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "896b18a03ab04ea6a056c9a62b3995ca"
format_stamp: "Formatted at 2026-09-25 15:03:54 on dist-test-slave-qxh0"
server_key: "7a0e76892633f824b447d97fe6905255"
server_key_iv: "0d461338e940b37bdbb2061ec779dd1f"
server_key_version: "encryptionkey@0"
I20260925 15:03:54.725639 27439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:03:54.728907 27454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:54.730154 27439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:03:54.730319 27439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "896b18a03ab04ea6a056c9a62b3995ca"
format_stamp: "Formatted at 2026-09-25 15:03:54 on dist-test-slave-qxh0"
server_key: "7a0e76892633f824b447d97fe6905255"
server_key_iv: "0d461338e940b37bdbb2061ec779dd1f"
server_key_version: "encryptionkey@0"
I20260925 15:03:54.730533 27439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:54.760244 27439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:54.763880 27439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:54.764130 27439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:54.781601 27439 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:41435
I20260925 15:03:54.781639 27505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:41435 every 8 connection(s)
I20260925 15:03:54.782884 27439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:54.786502 27506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:54.790066 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27439
I20260925 15:03:54.790269 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:54.790912 5071 external_mini_cluster.cc:1428] Setting key 50245ca30c19d20e9e6df355ccba787f
I20260925 15:03:54.793612 27506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:03:54 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.253@KRBTEST.COM" created.
I20260925 15:03:54.816859 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.796843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.4.243.254:41435)
0925 15:03:54.798564 (+ 1721us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:54.798573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:54.798670 (+ 97us) server_negotiation.cc:407] Connection header received
0925 15:03:54.800355 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:54.800361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:54.800599 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:54.800852 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:54.802264 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:54.803625 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:54.805107 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:54.805689 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:54.805899 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:54.809366 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:54.809374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:54.809386 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:54.809414 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:54.812317 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:54.813083 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:54.813092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:54.813095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:54.813244 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:54.813652 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:54.813660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:54.813664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:54.814160 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:54.814381 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:54.814984 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 15:03:54.815349 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1507,"thread_start_us":1286,"threads_started":1}
I20260925 15:03:54.816895 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.796843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:41435 (local address 127.0.0.1:49604)
0925 15:03:54.797741 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:54.797788 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:54.799951 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:54.800867 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:54.800874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:54.801665 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:54.802021 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:54.802038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:54.803789 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:54.803796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:54.804958 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:54.804970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:54.805934 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:54.805942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:54.806142 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:54.806860 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:54.806883 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:54.809150 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:54.812470 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:54.812483 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:54.812496 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:54.812910 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:54.813371 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:54.813379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:54.813383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:54.813544 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:54.814304 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:54.814312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:54.814545 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 15:03:54.814868 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 15:03:54.815464 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":144,"thread_start_us":371,"threads_started":1}
I20260925 15:03:54.819970 27506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:54.820989 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.820618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54114)
0925 15:03:54.820773 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:54.820875 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:03:54.821240 27457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
I20260925 15:03:54.822747 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.822373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54126)
0925 15:03:54.822523 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:54.822584 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:03:54.822993 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:45939: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab.
Entry for principal kudu/127.4.243.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab.
I20260925 15:03:54.843601 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:45939
--webserver_interface=127.4.243.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab
--principal=kudu/127.4.243.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:41435,127.4.243.253:45939,127.4.243.252:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:03:54.872330 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } attempt: 1
I20260925 15:03:54.873566 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.873069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54136)
0925 15:03:54.873285 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:54.873417 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:03:54.874661 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.874256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54138)
0925 15:03:54.874403 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:54.874476 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:03:54.874862 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:45939: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
I20260925 15:03:54.943111 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } attempt: 2
I20260925 15:03:54.944216 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.943751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54152)
0925 15:03:54.943986 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:54.944073 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:03:54.945201 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:54.944852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54166)
0925 15:03:54.945002 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:54.945076 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:03:54.945397 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:45939: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
I20260925 15:03:55.036648 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } attempt: 3
I20260925 15:03:55.037789 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.037332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54174)
0925 15:03:55.037573 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.037660 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 15:03:55.038829 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.038431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54178)
0925 15:03:55.038584 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.038704 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:03:55.039041 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:45939: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
W20260925 15:03:55.166452 27513 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:03:55.166827 27513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:55.166951 27513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:55.167079 27513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:55.174022 27513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:55.174115 27513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:55.174203 27513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:55.174276 27513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:55.174365 27513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:55.182296 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } attempt: 4
I20260925 15:03:55.183491 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.182963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54186)
0925 15:03:55.183271 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.183353 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260925 15:03:55.183969 27513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:41435,127.4.243.253:45939,127.4.243.252:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab
--principal=kudu/127.4.243.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.243.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:55.184546 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.184199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54192)
0925 15:03:55.184342 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.184424 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:03:55.184738 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:45939: Network error: Client connection negotiation failed: client connection to 127.4.243.253:45939: connect: Connection refused (error 111)
I20260925 15:03:55.185451 27513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:55.186801 27513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:55.198628 27521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:55.198674 27519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:55.198901 27518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:55.199210 27513 server_base.cc:1034] running on GCE node
I20260925 15:03:55.199922 27513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:55.201167 27513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:55.202384 27513 hybrid_clock.cc:648] HybridClock initialized: now 1790348635202328 us; error 34 us; skew 500 ppm
I20260925 15:03:55.202739 27513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:55.206190 27513 init.cc:377] Logged in from keytab as kudu/127.4.243.253@KRBTEST.COM (short username kudu)
I20260925 15:03:55.207553 27513 webserver.cc:460] Webserver started at http://127.4.243.253:35483/ using document root <none> and password file <none>
I20260925 15:03:55.208194 27513 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:55.208309 27513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:55.208606 27513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:55.211495 27513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data/instance:
uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65"
format_stamp: "Formatted at 2026-09-25 15:03:55 on dist-test-slave-qxh0"
server_key: "4890209b628327cfdf73e424425564a5"
server_key_iv: "08392acbed0bfed135405c920367658f"
server_key_version: "encryptionkey@0"
I20260925 15:03:55.212265 27513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/wal/instance:
uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65"
format_stamp: "Formatted at 2026-09-25 15:03:55 on dist-test-slave-qxh0"
server_key: "4890209b628327cfdf73e424425564a5"
server_key_iv: "08392acbed0bfed135405c920367658f"
server_key_version: "encryptionkey@0"
I20260925 15:03:55.217247 27513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 15:03:55.220618 27528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:55.221866 27513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:03:55.222030 27513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/wal
uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65"
format_stamp: "Formatted at 2026-09-25 15:03:55 on dist-test-slave-qxh0"
server_key: "4890209b628327cfdf73e424425564a5"
server_key_iv: "08392acbed0bfed135405c920367658f"
server_key_version: "encryptionkey@0"
I20260925 15:03:55.222234 27513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:55.244906 27513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:55.248929 27513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:55.249189 27513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:55.267342 27513 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.253:45939
I20260925 15:03:55.267361 27579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.253:45939 every 8 connection(s)
I20260925 15:03:55.268666 27513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
I20260925 15:03:55.272475 27580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:55.277207 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27513
I20260925 15:03:55.277382 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-1/wal/instance
I20260925 15:03:55.277905 5071 external_mini_cluster.cc:1428] Setting key 62ba0ab148a90de5f559ce0e687f4e8f
I20260925 15:03:55.279759 27580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:55.302444 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.283024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.4.243.254:41435)
0925 15:03:55.283293 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.283301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.284440 (+ 1139us) server_negotiation.cc:407] Connection header received
0925 15:03:55.287302 (+ 2862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.287306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.287374 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.287452 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.288980 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.290154 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.291673 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.292387 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.292522 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.296196 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.296201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.296204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.296235 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.299290 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.300221 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.300225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.300227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.300337 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.300796 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.300798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.300800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.301059 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.301242 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.302063 (+ 821us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.302222 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:03:55.303891 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.283449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:41435 (local address 127.0.0.1:49616)
0925 15:03:55.284264 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.284318 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.286981 (+ 2663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.287704 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.287717 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.288295 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.288814 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.288830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.290311 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.290315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.291530 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.291543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.292581 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.292586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.292769 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.293436 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.293457 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.295964 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:55.299420 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.299432 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.299450 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.300105 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.300439 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.300447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.300454 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.300686 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.301184 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:55.301195 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:55.301670 (+ 475us) client_negotiation.cc:769] Sending connection context
0925 15:03:55.301971 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 15:03:55.302483 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":296,"threads_started":1}
WARNING: no policy specified for kudu/127.4.243.252@KRBTEST.COM; defaulting to no policy
I20260925 15:03:55.305616 27580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.252@KRBTEST.COM" created.
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:03:55.321441 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.306478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.4.243.253:45939)
0925 15:03:55.307165 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.307172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.307204 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:03:55.307261 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.307269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.307509 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.307681 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.308396 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.310135 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.311338 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.312270 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.312406 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.315704 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.315720 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.315742 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.315790 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.318508 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.319345 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.319352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.319354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.319460 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.319868 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.319871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.319872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.320189 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.320341 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.320801 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.321140 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":510,"thread_start_us":366,"threads_started":1}
I20260925 15:03:55.321475 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.306422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:54200)
0925 15:03:55.306637 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.306657 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.306762 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.307693 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.307696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.308084 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.308214 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.308222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.310282 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.310286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.311211 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.311222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.312447 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.312451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.312578 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.313217 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.313234 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.315485 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:55.318694 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.318701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.318708 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.319226 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.319567 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.319575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.319580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.319756 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.320306 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:55.320314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:55.320486 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 15:03:55.320631 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:03:55.320980 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 15:03:55.324388 27580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:55.325652 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.325167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:38368)
0925 15:03:55.325391 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.325508 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 15:03:55.326042 27530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.327610 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.327320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:38382)
0925 15:03:55.327443 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.327517 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:03:55.327900 27580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab.
Entry for principal kudu/127.4.243.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab.
I20260925 15:03:55.341455 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:33221
--webserver_interface=127.4.243.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab
--principal=kudu/127.4.243.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:41435,127.4.243.253:45939,127.4.243.252:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:03:55.377243 27580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 1
I20260925 15:03:55.378436 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.377961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:38392)
0925 15:03:55.378183 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.378286 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:03:55.379472 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.379101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:38400)
0925 15:03:55.379269 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.379351 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:03:55.379706 27580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.447932 27580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 2
I20260925 15:03:55.449074 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.448619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53764)
0925 15:03:55.448817 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.448913 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 15:03:55.450085 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.449751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53778)
0925 15:03:55.449892 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.449954 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:03:55.450285 27580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.484015 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } attempt: 5
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:03:55.498683 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.484777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:33922)
0925 15:03:55.484980 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.485003 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.485120 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.485464 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.485468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.485841 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.486039 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.486050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.487463 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.487468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.488345 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.488357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.489484 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.489488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.489674 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.490351 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.490369 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.492392 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:55.496473 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.496477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.496480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.496858 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.497235 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.497239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.497241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.497363 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.497927 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:55.497930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:55.498058 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 15:03:55.498183 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:03:55.498377 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:03:55.498869 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.484884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.4.243.253:45939)
0925 15:03:55.485092 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.485098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.485123 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:03:55.485218 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.485224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.485304 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.485426 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.486201 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.487309 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.488507 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.489327 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.489483 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.492622 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.492631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.492637 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.492687 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.496339 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.496975 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.496981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.496985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.497125 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.497456 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.497461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.497464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.497820 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.497968 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.498319 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.498550 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:03:55.499989 27506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:55.501123 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.500751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53794)
0925 15:03:55.500897 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.500967 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:03:55.502072 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.501784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53804)
0925 15:03:55.501921 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.501981 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:03:55.502278 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.541498 27580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 3
I20260925 15:03:55.542639 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.542119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53810)
0925 15:03:55.542338 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.542424 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:03:55.543689 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.543301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53812)
0925 15:03:55.543509 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.543585 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260925 15:03:55.543888 27580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.553493 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 1
I20260925 15:03:55.554697 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.554122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53826)
0925 15:03:55.554358 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.554467 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 15:03:55.555832 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.555355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53830)
0925 15:03:55.555562 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.555674 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:03:55.556098 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.624327 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 2
I20260925 15:03:55.625653 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.625064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53834)
0925 15:03:55.625305 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.625420 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:03:55.626775 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.626360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53846)
0925 15:03:55.626515 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.626636 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:03:55.627030 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
W20260925 15:03:55.669407 27587 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:03:55.669726 27587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:55.669795 27587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:55.669876 27587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:55.676959 27587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:55.677043 27587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:55.677104 27587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:55.677151 27587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:55.677208 27587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:55.686472 27587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:41435,127.4.243.253:45939,127.4.243.252:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab
--principal=kudu/127.4.243.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.243.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:55.687081 27580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 4
I20260925 15:03:55.687845 27587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:55.688179 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.687726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53860)
0925 15:03:55.687939 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.688038 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:03:55.689067 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.688782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53874)
0925 15:03:55.688921 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.688978 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:03:55.689296 27580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.689334 27587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:55.700166 27592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:55.700192 27593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:55.700179 27595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:55.701224 27587 server_base.cc:1034] running on GCE node
I20260925 15:03:55.701869 27587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:55.703217 27587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:55.704422 27587 hybrid_clock.cc:648] HybridClock initialized: now 1790348635704345 us; error 49 us; skew 500 ppm
I20260925 15:03:55.704735 27587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:55.707885 27587 init.cc:377] Logged in from keytab as kudu/127.4.243.252@KRBTEST.COM (short username kudu)
I20260925 15:03:55.709337 27587 webserver.cc:460] Webserver started at http://127.4.243.252:36275/ using document root <none> and password file <none>
I20260925 15:03:55.710016 27587 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:55.710134 27587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:55.710457 27587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:55.713387 27587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data/instance:
uuid: "7bf49847988344b39ad76cbfe76cb748"
format_stamp: "Formatted at 2026-09-25 15:03:55 on dist-test-slave-qxh0"
server_key: "a4c4b0e80223779c8a2925a229bb4476"
server_key_iv: "ca3a271049fb78f2b68a89d4864c0f32"
server_key_version: "encryptionkey@0"
I20260925 15:03:55.714162 27587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/wal/instance:
uuid: "7bf49847988344b39ad76cbfe76cb748"
format_stamp: "Formatted at 2026-09-25 15:03:55 on dist-test-slave-qxh0"
server_key: "a4c4b0e80223779c8a2925a229bb4476"
server_key_iv: "ca3a271049fb78f2b68a89d4864c0f32"
server_key_version: "encryptionkey@0"
I20260925 15:03:55.719148 27587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 15:03:55.722306 27602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:55.723682 27587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:55.723913 27587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/wal
uuid: "7bf49847988344b39ad76cbfe76cb748"
format_stamp: "Formatted at 2026-09-25 15:03:55 on dist-test-slave-qxh0"
server_key: "a4c4b0e80223779c8a2925a229bb4476"
server_key_iv: "ca3a271049fb78f2b68a89d4864c0f32"
server_key_version: "encryptionkey@0"
I20260925 15:03:55.724134 27587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:55.733230 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 3
I20260925 15:03:55.734241 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.733751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53880)
0925 15:03:55.733956 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.734054 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:03:55.735316 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.734917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53892)
0925 15:03:55.735116 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.735199 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 15:03:55.735512 27506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:33221: Network error: Client connection negotiation failed: client connection to 127.4.243.252:33221: connect: Connection refused (error 111)
I20260925 15:03:55.742122 27587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:55.745707 27587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:55.745909 27587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:55.763765 27587 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.252:33221
I20260925 15:03:55.763787 27653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.252:33221 every 8 connection(s)
I20260925 15:03:55.765079 27587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
I20260925 15:03:55.768654 27654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:55.774928 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27587
I20260925 15:03:55.775116 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284481048-5071-0/minicluster-data/master-2/wal/instance
I20260925 15:03:55.774570 27654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:55.775576 5071 external_mini_cluster.cc:1428] Setting key 8eee9ac228095db6a0030f8803916e5c
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:03:55.795800 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.777601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.4.243.254:41435)
0925 15:03:55.777829 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.777837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.778381 (+ 544us) server_negotiation.cc:407] Connection header received
0925 15:03:55.780049 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.780055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.780139 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.780237 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.781864 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.782958 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.784393 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.784964 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.785096 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.789406 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.789412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.789416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.789456 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.792512 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.793230 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.793236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.793239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.793370 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.793793 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.793798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.793801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.794179 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.794285 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.795179 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.795421 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:03:55.796932 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.777389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.4.243.252:33221)
0925 15:03:55.778176 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.778186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.778224 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:03:55.779785 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.779806 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.780073 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.780277 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.781929 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.783110 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.784750 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.785486 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.785622 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.789686 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.789696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.789711 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.789752 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.792776 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.793422 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.793453 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.793458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.793653 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.794046 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.794049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.794051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.794825 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.795141 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.795468 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.795674 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":614,"spinlock_wait_cycles":18688,"thread_start_us":418,"threads_started":1}
I20260925 15:03:55.796952 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.777388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:41435 (local address 127.0.0.1:39166)
0925 15:03:55.778224 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.778269 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.779797 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.780312 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.780318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.781141 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.781675 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.781704 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.783116 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.783121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.784238 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.784249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.785236 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.785244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.785495 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.786319 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.786347 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.789182 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:55.792662 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.792674 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.792686 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.793115 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.793488 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.793492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.793498 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.793687 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.794301 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:55.794307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:55.794799 (+ 492us) client_negotiation.cc:769] Sending connection context
0925 15:03:55.795079 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 15:03:55.795560 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":22,"thread_start_us":402,"threads_started":1}
I20260925 15:03:55.798452 27654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:03:55.811759 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.799276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:45939 (local address 127.0.0.1:33930)
0925 15:03:55.799514 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.799528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.799653 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.800135 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.800141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.800573 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.800776 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.800790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.801749 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.801752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.802748 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.802762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.803391 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.803396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.803584 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.804181 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.804197 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.806380 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:55.809451 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.809457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.809461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.809801 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.810192 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.810197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.810200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.810330 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.810841 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:55.810845 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:55.810990 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:03:55.811144 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 15:03:55.811415 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 15:03:55.811790 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.799237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.4.243.253:45939)
0925 15:03:55.799429 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.799437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.799599 (+ 162us) server_negotiation.cc:407] Connection header received
0925 15:03:55.799769 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.799773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.799855 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.799949 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.800933 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.801599 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.802874 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.803263 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.803356 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.806553 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.806559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.806563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.806636 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.809323 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.809915 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.809921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.809924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.810067 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.810431 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.810437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.810440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.810733 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.810873 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.811242 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.811442 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:03:55.812788 27654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:55.813488 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.800010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.4.243.254:41435)
0925 15:03:55.800169 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.800173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.800189 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:03:55.800357 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.800360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.800412 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.800485 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.801681 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.802374 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.803879 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.804372 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.804464 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.808331 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.808335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.808338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.808365 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.810851 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.811481 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.811487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.811490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.811620 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.812045 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.812052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.812055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.812399 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.812671 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.812958 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.813161 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:03:55.825125 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.813611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.4.243.252:33221)
0925 15:03:55.813735 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.813740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.813757 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:03:55.813815 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.813819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.813881 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.813959 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.814690 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.815662 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.816751 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.817174 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.817265 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.820303 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.820307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.820310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.820336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.822892 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.823441 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.823449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.823453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.823563 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.823878 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.823885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.823888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.824123 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.824259 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.824566 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.824763 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:03:55.825246 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.813461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53908)
0925 15:03:55.813629 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.813647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.813730 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.814008 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.814012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.814375 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.814530 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.814544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.815794 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.815798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.816625 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.816635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.817306 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.817312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.817506 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.818046 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.818081 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.820145 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:55.823007 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.823011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.823013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.823335 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.823661 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.823668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.823671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.823786 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.824224 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:55.824229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:55.824355 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 15:03:55.824489 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 15:03:55.824893 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":137}
I20260925 15:03:55.829097 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.815322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.4.243.253:45939)
0925 15:03:55.815459 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.815463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.815476 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:03:55.815599 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.815602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.815648 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.815717 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.816809 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.817603 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.818985 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.819406 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.819543 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.823460 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.823464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.823467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.823493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.826445 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.827169 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.827174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.827179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.827329 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.827748 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.827753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.827756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.828090 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.828346 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.828510 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.828762 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:03:55.831435 27654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: Bootstrap starting.
I20260925 15:03:55.833782 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:03:55.835397 27654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:55.836696 27654 log.cc:824] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:55.839353 27654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: No bootstrap required, opened a new log
I20260925 15:03:55.837160 27481 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "896b18a03ab04ea6a056c9a62b3995ca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39182
I20260925 15:03:55.845842 27654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:55.846160 27654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf49847988344b39ad76cbfe76cb748, State: Initialized, Role: FOLLOWER
I20260925 15:03:55.846571 27654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:55.847525 27660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:55.847922 27660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:03:55.848328 27654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:03:55.856608 27671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:03:55.856727 27671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:03:55.854951 27587 master_runner.cc:186] Error getting master registration for 127.4.243.254:41435: OK, instance_id { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" instance_seqno: 1790348634776713 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:03:55.865610 27587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:03:55.912709 27506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 4
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348634, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:03:55.930302 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.913509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53912)
0925 15:03:55.913753 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.913776 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.913945 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.914338 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.914343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.914897 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.915117 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.915128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.916155 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.916161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.917119 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.917131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.917884 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.917889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.918081 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.918821 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.918850 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.921888 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:55.926964 (+ 5076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.926974 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.926980 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.927511 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.928076 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:55.928081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:55.928085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:55.928260 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:55.929304 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:55.929311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:55.929575 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 15:03:55.929722 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:03:55.930025 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 15:03:55.930495 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.913539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.4.243.252:33221)
0925 15:03:55.913755 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.913764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.913842 (+ 78us) server_negotiation.cc:407] Connection header received
0925 15:03:55.914106 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.914112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.914198 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.914300 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.915266 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.916005 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.917255 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.917734 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.917857 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.922088 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.922095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.922100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.922139 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:55.925735 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.927689 (+ 1954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.927694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.927698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.927853 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:55.928418 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:55.928425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:55.928461 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:55.929143 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:55.929358 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:55.929871 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 15:03:55.930129 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:03:55.935114 27506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca: Bootstrap starting.
I20260925 15:03:55.938133 27506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:55.939414 27506 log.cc:824] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:55.941906 27506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca: No bootstrap required, opened a new log
I20260925 15:03:55.946951 27506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:55.947302 27506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896b18a03ab04ea6a056c9a62b3995ca, State: Initialized, Role: FOLLOWER
I20260925 15:03:55.947732 27506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:55.948771 27674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:55.949123 27674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:03:55.949937 27506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:55.950122 27481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:03:55.950333 27481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:55.954610 27481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:55.956707 27481 leader_election.cc:288] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [CANDIDATE]: Term 1 election: Requested vote from peers 3ed48f4181a04ce7aaa8a04fa84b8c65 (127.4.243.253:45939), 7bf49847988344b39ad76cbfe76cb748 (127.4.243.252:33221)
I20260925 15:03:55.958204 27629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "896b18a03ab04ea6a056c9a62b3995ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7bf49847988344b39ad76cbfe76cb748"
I20260925 15:03:55.958843 27629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:55.963523 27629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896b18a03ab04ea6a056c9a62b3995ca in term 1.
I20260925 15:03:55.964434 27456 leader_election.cc:302] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf49847988344b39ad76cbfe76cb748, 896b18a03ab04ea6a056c9a62b3995ca; no voters:
I20260925 15:03:55.965075 27674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:03:55.969501 27674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [term 1 LEADER]: Becoming Leader. State: Replica: 896b18a03ab04ea6a056c9a62b3995ca, State: Running, Role: LEADER
I20260925 15:03:55.970185 27674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:55.978583 27682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896b18a03ab04ea6a056c9a62b3995ca. Latest consensus state: current_term: 1 leader_uuid: "896b18a03ab04ea6a056c9a62b3995ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:55.979153 27682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:55.985407 27688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:03:55.988490 27580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } attempt: 5
W20260925 15:03:55.997859 27608 tablet.cc:2369] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:03:55 dist-test-slave-qxh0 krb5kdc[27426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348635, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:03:56.001968 27688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:03:56.005986 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.989131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:33221 (local address 127.0.0.1:53922)
0925 15:03:55.989541 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:55.989561 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:55.989688 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:55.990238 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:55.990243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:55.990792 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:55.991007 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.991019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.992627 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.992632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.994078 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:55.994093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.995173 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.995179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:55.995416 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.996059 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:55.996084 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:55.998829 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:56.003137 (+ 4308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:56.003141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:56.003145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:56.003640 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:56.004102 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:56.004107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:56.004111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:56.004281 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:56.004960 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:56.004967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:56.005160 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 15:03:56.005318 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 15:03:56.005587 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260925 15:03:56.006968 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.989500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.4.243.252:33221)
0925 15:03:55.989703 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.989709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:55.989733 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:03:55.989801 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:55.989806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:55.989906 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:55.990016 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:55.991275 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.992425 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.994259 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:55.995028 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:55.995159 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:55.999033 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:55.999039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:55.999043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:55.999095 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:56.002968 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.003777 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.003782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.003785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.003942 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.004411 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.004416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.004419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.004760 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:56.004894 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.006362 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.006611 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:03:56.014151 27580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65: Bootstrap starting.
W20260925 15:03:56.016760 27460 tablet.cc:2369] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:03:56.019234 27580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65: Neither blocks nor log segments found. Creating new log.
I20260925 15:03:56.020835 27580 log.cc:824] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65: Log is configured to *not* fsync() on all Append() calls
I20260925 15:03:56.031188 27580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65: No bootstrap required, opened a new log
I20260925 15:03:56.053141 27629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [term 1 FOLLOWER]: Refusing update from remote peer 896b18a03ab04ea6a056c9a62b3995ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:56.054149 27682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:56.054656 27439 master_runner.cc:186] Error getting master registration for 127.4.243.253:45939: OK, instance_id { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" instance_seqno: 1790348635262202 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:03:56.066689 27439 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:03:56.075851 27660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896b18a03ab04ea6a056c9a62b3995ca. Latest consensus state: current_term: 1 leader_uuid: "896b18a03ab04ea6a056c9a62b3995ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:56.076179 27660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:03:56.078872 27696 mvcc.cc:205] Tried to move back new op lower bound from 7333268013205172224 to 7333268012940673024. Current Snapshot: MvccSnapshot[applied={T|T < 7333268013205172224}]
I20260925 15:03:56.079246 27674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896b18a03ab04ea6a056c9a62b3995ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:56.079563 27674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:56.080823 27700 mvcc.cc:205] Tried to move back new op lower bound from 7333268013205172224 to 7333268012940673024. Current Snapshot: MvccSnapshot[applied={T|T < 7333268013205172224}]
I20260925 15:03:56.083777 27688 catalog_manager.cc:1347] Generated new cluster ID: 83fb70953a934cada4007c0719d2bb21
I20260925 15:03:56.083875 27688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:03:56.088876 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.997346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.4.243.253:45939)
0925 15:03:55.997527 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:55.997533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.033240 (+ 35707us) server_negotiation.cc:407] Connection header received
0925 15:03:56.033666 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.033671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.033749 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.033819 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.046716 (+ 12897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.047925 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.061030 (+ 13105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.061693 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.061794 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.076719 (+ 14925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:56.076728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:56.076737 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:56.076784 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:56.082206 (+ 5422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.085048 (+ 2842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.085087 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.085090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.085371 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.086169 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.086174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.086178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.086751 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:56.087095 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.087932 (+ 837us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.088314 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:03:56.094707 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:55.999491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.4.243.252:33221)
0925 15:03:56.007217 (+ 7726us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.007222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.041933 (+ 34711us) server_negotiation.cc:407] Connection header received
0925 15:03:56.042312 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.042319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.042405 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.042476 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.053876 (+ 11400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.054997 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.066995 (+ 11998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.067723 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.067863 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.076999 (+ 9136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:56.077006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:56.077011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:56.077051 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:56.084173 (+ 7122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.086319 (+ 2146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.086325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.086330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.086470 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.092037 (+ 5567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.092043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.092047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.092408 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:56.092654 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.093526 (+ 872us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.093931 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7623,"thread_start_us":37783,"threads_started":1}
I20260925 15:03:56.097900 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.027295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.4.243.254:41435)
0925 15:03:56.027477 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.027483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.044389 (+ 16906us) server_negotiation.cc:407] Connection header received
0925 15:03:56.044964 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.044970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.045052 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.045122 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.058718 (+ 13596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.059867 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.069016 (+ 9149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.069769 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.069900 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.086993 (+ 17093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:56.087001 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:56.087007 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:56.087066 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:56.091077 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.093777 (+ 2700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.093782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.093787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.093951 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:56.094450 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:56.094456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:56.094459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:56.095751 (+ 1292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:56.096010 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.096673 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.097144 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:03:56.102756 27580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:56.103193 27580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed48f4181a04ce7aaa8a04fa84b8c65, State: Initialized, Role: FOLLOWER
I20260925 15:03:56.103725 27580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } }
I20260925 15:03:56.106796 27660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896b18a03ab04ea6a056c9a62b3995ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:56.107064 27660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:03:56.110903 27704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:56.111332 27704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:03:56.113286 27580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:03:56.115007 27555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "896b18a03ab04ea6a056c9a62b3995ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65"
I20260925 15:03:56.115585 27555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:03:56.120569 27555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896b18a03ab04ea6a056c9a62b3995ca in term 1.
I20260925 15:03:56.121598 27554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [term 1 FOLLOWER]: Refusing update from remote peer 896b18a03ab04ea6a056c9a62b3995ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:03:56.123849 27682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:03:56.148535 27688 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:03:56.151734 27709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "896b18a03ab04ea6a056c9a62b3995ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:56.152122 27704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896b18a03ab04ea6a056c9a62b3995ca. Latest consensus state: current_term: 1 leader_uuid: "896b18a03ab04ea6a056c9a62b3995ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:56.152356 27704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:03:56.152385 27709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:03:56.153723 27674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "896b18a03ab04ea6a056c9a62b3995ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896b18a03ab04ea6a056c9a62b3995ca" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 41435 } } peers { permanent_uuid: "3ed48f4181a04ce7aaa8a04fa84b8c65" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 45939 } } peers { permanent_uuid: "7bf49847988344b39ad76cbfe76cb748" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 33221 } } }
I20260925 15:03:56.154098 27674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca [sys.catalog]: This master's current role is: LEADER
I20260925 15:03:56.159461 27688 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:03:56.173651 27688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 896b18a03ab04ea6a056c9a62b3995ca: Generated new TSK 0
I20260925 15:03:56.174644 27688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:03:56.217252 27720 catalog_manager.cc:1259] Loaded cluster ID: 83fb70953a934cada4007c0719d2bb21
I20260925 15:03:56.217464 27720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65: loading cluster ID for follower catalog manager: success
I20260925 15:03:56.221427 27720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65: acquiring CA information for follower catalog manager: success
I20260925 15:03:56.224035 27720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ed48f4181a04ce7aaa8a04fa84b8c65: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:03:56.320557 27584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.299646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.4.243.253:45939)
0925 15:03:56.299900 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.299907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.317143 (+ 17236us) server_negotiation.cc:407] Connection header received
0925 15:03:56.319042 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.319049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.319135 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.319207 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.320292 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33952: BlockingRecv error: recv got EOF from 127.0.0.1:33952 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 15:03:56.322470 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.301791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.4.243.252:33221)
0925 15:03:56.301985 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.301991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.318567 (+ 16576us) server_negotiation.cc:407] Connection header received
0925 15:03:56.321056 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.321063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.321150 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.321223 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.322241 (+ 1018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53944: BlockingRecv error: recv got EOF from 127.0.0.1:53944 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 15:03:56.325011 27509 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.315092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.4.243.254:41435)
0925 15:03:56.315324 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.315330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.323093 (+ 7763us) server_negotiation.cc:407] Connection header received
0925 15:03:56.323247 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.323252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.323331 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.323443 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.324757 (+ 1314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39198: BlockingRecv error: recv got EOF from 127.0.0.1:39198 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:03:56.372856 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.356713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.4.243.253:45939)
0925 15:03:56.356913 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.356919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.364684 (+ 7765us) server_negotiation.cc:407] Connection header received
0925 15:03:56.365261 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.365268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.365499 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.365692 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.366477 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.367711 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.369737 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.370403 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.370497 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.371243 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.371997 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.372123 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.372376 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.372479 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:03:56.419294 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.406196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.4.243.252:33221)
0925 15:03:56.406393 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.406399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.417108 (+ 10709us) server_negotiation.cc:407] Connection header received
0925 15:03:56.417618 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.417624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.417745 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.417819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.419036 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53958: BlockingRecv error: recv got EOF from 127.0.0.1:53958 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:03:56.469235 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.454983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.4.243.253:45939)
0925 15:03:56.455207 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.455213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.462431 (+ 7218us) server_negotiation.cc:407] Connection header received
0925 15:03:56.462919 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.462926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.463016 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.463079 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.463788 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.464944 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.466952 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.467673 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.467769 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.468356 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.468517 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.468579 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.468811 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.468896 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 15:03:56.485792 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.483687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.4.243.252:33221)
0925 15:03:56.483925 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.483935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.484427 (+ 492us) server_negotiation.cc:407] Connection header received
0925 15:03:56.484650 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.484661 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.484731 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.484840 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.485558 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53966: BlockingRecv error: recv got EOF from 127.0.0.1:53966 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:03:56.504004 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.498441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.4.243.253:45939)
0925 15:03:56.498743 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.498756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.499253 (+ 497us) server_negotiation.cc:407] Connection header received
0925 15:03:56.499468 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.499473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.499539 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.499644 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.500085 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.500858 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.502146 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.502571 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.502735 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.503214 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.503342 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.503424 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.503666 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.503746 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 15:03:56.512567 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.510903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.4.243.252:33221)
0925 15:03:56.511111 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.511117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.511138 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:03:56.511355 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.511361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.511450 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.511507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.512323 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53970: BlockingRecv error: recv got EOF from 127.0.0.1:53970 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:03:56.525053 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.519934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.4.243.253:45939)
0925 15:03:56.520172 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.520181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.520645 (+ 464us) server_negotiation.cc:407] Connection header received
0925 15:03:56.520838 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.520847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.520928 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.521021 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.521387 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.522115 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.523393 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.523839 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.523967 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.524324 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.524416 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.524504 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.524653 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.524735 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 15:03:56.536993 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.535349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.4.243.252:33221)
0925 15:03:56.535618 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.535625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.535644 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:03:56.535882 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.535889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.535968 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.536074 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.536813 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53972: BlockingRecv error: recv got EOF from 127.0.0.1:53972 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260925 15:03:56.554399 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.548380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.4.243.253:45939)
0925 15:03:56.548610 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.548619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.549133 (+ 514us) server_negotiation.cc:407] Connection header received
0925 15:03:56.549425 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.549429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.549498 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.549604 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.550061 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.551110 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.552652 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.553105 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.553248 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.553639 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.553749 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.553842 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.553969 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.554055 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 15:03:56.562487 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.560459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.4.243.252:33221)
0925 15:03:56.560711 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.560719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.561131 (+ 412us) server_negotiation.cc:407] Connection header received
0925 15:03:56.561322 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.561327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.561410 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.561513 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.562222 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53988: BlockingRecv error: recv got EOF from 127.0.0.1:53988 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:03:56.587455 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.581893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.4.243.253:45939)
0925 15:03:56.582117 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.582127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.582673 (+ 546us) server_negotiation.cc:407] Connection header received
0925 15:03:56.582941 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.582946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.583023 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.583125 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.583539 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.584378 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.585659 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.586124 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.586260 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.586714 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.586823 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.586923 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.587072 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.587158 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:03:56.598441 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.596333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.4.243.252:33221)
0925 15:03:56.596563 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.596569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.597082 (+ 513us) server_negotiation.cc:407] Connection header received
0925 15:03:56.597288 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.597293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.597375 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.597478 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.598207 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53996: BlockingRecv error: recv got EOF from 127.0.0.1:53996 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:03:56.641307 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.633888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.4.243.253:45939)
0925 15:03:56.634174 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.634184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.634789 (+ 605us) server_negotiation.cc:407] Connection header received
0925 15:03:56.635133 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.635139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.635249 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.635370 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.635907 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.637070 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.639035 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.639762 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.639895 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.640433 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.640593 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.640696 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.640868 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.640955 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 15:03:56.652789 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.650416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.4.243.252:33221)
0925 15:03:56.650673 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.650680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.651231 (+ 551us) server_negotiation.cc:407] Connection header received
0925 15:03:56.651504 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.651510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.651604 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.651707 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.652554 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54006: BlockingRecv error: recv got EOF from 127.0.0.1:54006 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:03:56.726159 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.720294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.4.243.253:45939)
0925 15:03:56.720487 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.720494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.720797 (+ 303us) server_negotiation.cc:407] Connection header received
0925 15:03:56.720972 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.720980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.721081 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.721187 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.721550 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.722790 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.724068 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.724689 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.724804 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.725274 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.725432 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.725512 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.725798 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.725875 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:03:56.733294 27658 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:03:56.731504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.4.243.252:33221)
0925 15:03:56.731711 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.731717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.731963 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:03:56.732142 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.732146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.732216 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.732296 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:56.733079 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54016: BlockingRecv error: recv got EOF from 127.0.0.1:54016 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:03:56.858729 27671 catalog_manager.cc:1259] Loaded cluster ID: 83fb70953a934cada4007c0719d2bb21
I20260925 15:03:56.858824 27671 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: loading cluster ID for follower catalog manager: success
I20260925 15:03:56.861172 27671 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: acquiring CA information for follower catalog manager: success
I20260925 15:03:56.863293 27671 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bf49847988344b39ad76cbfe76cb748: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:03:56.869582 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.864926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.4.243.253:45939)
0925 15:03:56.865197 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.865203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.865415 (+ 212us) server_negotiation.cc:407] Connection header received
0925 15:03:56.865591 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.865595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.865676 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.865751 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.866112 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.866845 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.868078 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.868459 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.868557 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.868892 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.868994 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.869057 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.869195 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.869256 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 15:03:56.880182 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.874966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.4.243.252:33221)
0925 15:03:56.875236 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.875243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.875499 (+ 256us) server_negotiation.cc:407] Connection header received
0925 15:03:56.875637 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.875642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.875783 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.875878 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.876232 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.876907 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.878109 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.878487 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.878580 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.879045 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.879641 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.879781 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.879818 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.879881 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 15:03:56.893654 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.885836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.4.243.252:33221)
0925 15:03:56.886278 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.886286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.887282 (+ 996us) server_negotiation.cc:407] Connection header received
0925 15:03:56.887479 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.887487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.887588 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.887688 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.888397 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.889490 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.891717 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.892344 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.892471 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.892966 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.893087 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.893175 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.893320 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.893389 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260925 15:03:56.893723 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.885735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.4.243.253:45939)
0925 15:03:56.886106 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.886114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.888095 (+ 1981us) server_negotiation.cc:407] Connection header received
0925 15:03:56.888310 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.888315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.888405 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.888490 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.889264 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.890107 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.891982 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.892426 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.892546 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.893030 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.893120 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.893214 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.893401 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.893487 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260925 15:03:56.895153 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:56.885662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.4.243.254:41435)
0925 15:03:56.886645 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:56.886650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:56.888823 (+ 2173us) server_negotiation.cc:407] Connection header received
0925 15:03:56.889064 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:56.889070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:56.889266 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:56.889441 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:03:56.890123 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.890880 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.892664 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:56.893214 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:56.893329 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:56.894076 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:03:56.894504 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:03:56.894571 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:56.894805 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:03:56.894883 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":759,"threads_started":1}
I20260925 15:03:56.899400 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27439
I20260925 15:03:56.900434 27501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:57.010849 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27439
I20260925 15:03:57.031903 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27513
I20260925 15:03:57.033126 27575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:57.141369 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27513
I20260925 15:03:57.161535 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27587
I20260925 15:03:57.162712 27649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:03:57.267907 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27587
2026-09-25T15:03:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4232 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:03:57 dist-test-slave-qxh0 krb5kdc[27835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:03:57 dist-test-slave-qxh0 krb5kdc[27835](info): set up 2 sockets
Sep 25 15:03:57 dist-test-slave-qxh0 krb5kdc[27835](info): commencing operation
krb5kdc: starting...
W20260925 15:03:58.359551 5071 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/test-user.keytab.
Sep 25 15:03:58 dist-test-slave-qxh0 krb5kdc[27835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:03:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.243.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
Entry for principal kudu/127.4.243.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab.
I20260925 15:03:58.601159 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35159
--webserver_interface=127.4.243.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35159,127.4.243.253:32827,127.4.243.252:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:03:58.919567 27848 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:03:58.919911 27848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:58.920023 27848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:58.920151 27848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:58.927204 27848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:58.927307 27848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:58.927431 27848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:58.927507 27848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:58.927611 27848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:58.937575 27848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35159,127.4.243.253:32827,127.4.243.252:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.254.keytab
--principal=kudu/127.4.243.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.254:35159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.243.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:58.939355 27848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:58.940819 27848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:58.952227 27853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:58.952224 27856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:58.952296 27854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:58.953123 27848 server_base.cc:1034] running on GCE node
I20260925 15:03:58.953861 27848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:58.955314 27848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:58.956573 27848 hybrid_clock.cc:648] HybridClock initialized: now 1790348638956505 us; error 52 us; skew 500 ppm
I20260925 15:03:58.956898 27848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:58 dist-test-slave-qxh0 krb5kdc[27835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:58.963141 27848 init.cc:377] Logged in from keytab as kudu/127.4.243.254@KRBTEST.COM (short username kudu)
I20260925 15:03:58.964459 27848 webserver.cc:460] Webserver started at http://127.4.243.254:45309/ using document root <none> and password file <none>
I20260925 15:03:58.965139 27848 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:58.965266 27848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:58.965571 27848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:58.968490 27848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data/instance:
uuid: "db5d2a45e55642cfa4dc1cfddff490e4"
format_stamp: "Formatted at 2026-09-25 15:03:58 on dist-test-slave-qxh0"
server_key: "af574dcb9b288dc4ef6d26bc94397fcc"
server_key_iv: "ff7dddce65bc8a9b413dea4adbd73f5e"
server_key_version: "encryptionkey@0"
I20260925 15:03:58.969280 27848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/wal/instance:
uuid: "db5d2a45e55642cfa4dc1cfddff490e4"
format_stamp: "Formatted at 2026-09-25 15:03:58 on dist-test-slave-qxh0"
server_key: "af574dcb9b288dc4ef6d26bc94397fcc"
server_key_iv: "ff7dddce65bc8a9b413dea4adbd73f5e"
server_key_version: "encryptionkey@0"
I20260925 15:03:58.974366 27848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:03:58.977646 27863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:58.979002 27848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:03:58.979190 27848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/wal
uuid: "db5d2a45e55642cfa4dc1cfddff490e4"
format_stamp: "Formatted at 2026-09-25 15:03:58 on dist-test-slave-qxh0"
server_key: "af574dcb9b288dc4ef6d26bc94397fcc"
server_key_iv: "ff7dddce65bc8a9b413dea4adbd73f5e"
server_key_version: "encryptionkey@0"
I20260925 15:03:58.979394 27848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:59.000835 27848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:59.005003 27848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:59.005285 27848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:59.023561 27848 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.254:35159
I20260925 15:03:59.023597 27914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.254:35159 every 8 connection(s)
I20260925 15:03:59.024833 27848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/data/info.pb
I20260925 15:03:59.028378 27915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:59.033895 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27848
I20260925 15:03:59.034094 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-0/wal/instance
I20260925 15:03:59.034660 5071 external_mini_cluster.cc:1428] Setting key 857d67e1b102a7eec5470c96be1355e6
I20260925 15:03:59.035496 27915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:03:59 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.253@KRBTEST.COM; defaulting to no policy
I20260925 15:03:59.060706 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.039495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.4.243.254:35159)
0925 15:03:59.041386 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:59.041392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:59.041436 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:03:59.043763 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:59.043769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:59.044003 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:59.044175 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:59.045585 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.047037 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.049263 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.049830 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.049955 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.053862 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:59.053870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:59.053881 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:59.053908 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:59.056624 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.057276 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.057288 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.057291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.057423 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.057778 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.057785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.057789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.058223 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:59.058418 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:59.058940 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 15:03:59.059299 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1702,"spinlock_wait_cycles":40192,"thread_start_us":1467,"threads_started":1}
I20260925 15:03:59.060688 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.039495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35159 (local address 127.0.0.1:55826)
0925 15:03:59.040393 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.040449 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:59.043358 (+ 2909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:59.044242 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:59.044252 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:59.045014 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:59.045367 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.045383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.047186 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.047195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.049118 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.049135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.050068 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.050077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.050296 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.051031 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:59.051063 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:59.053631 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:59.056754 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.056766 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.056778 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.057158 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.057523 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.057529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.057533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.057680 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.058345 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:59.058354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:59.058562 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 15:03:59.058853 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 15:03:59.059405 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":153,"thread_start_us":361,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.253@KRBTEST.COM" created.
I20260925 15:03:59.063700 27915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:59.064675 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.064305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52158)
0925 15:03:59.064461 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.064566 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:03:59.064881 27864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
I20260925 15:03:59.066329 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.065985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52168)
0925 15:03:59.066141 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.066205 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:03:59.066532 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32827: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab.
Entry for principal kudu/127.4.243.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab.
I20260925 15:03:59.093120 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:32827
--webserver_interface=127.4.243.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab
--principal=kudu/127.4.243.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35159,127.4.243.253:32827,127.4.243.252:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:03:59.115878 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } attempt: 1
I20260925 15:03:59.117101 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.116537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52176)
0925 15:03:59.116764 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.116912 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:03:59.118067 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.117700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52178)
0925 15:03:59.117836 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.117916 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:03:59.118294 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32827: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
I20260925 15:03:59.186571 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } attempt: 2
I20260925 15:03:59.188057 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.187404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52194)
0925 15:03:59.187730 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.187877 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:03:59.189146 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.188764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52204)
0925 15:03:59.188924 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.189003 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:03:59.189378 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32827: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
I20260925 15:03:59.280666 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } attempt: 3
I20260925 15:03:59.282033 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.281377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52212)
0925 15:03:59.281680 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.281824 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 15:03:59.283456 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.282894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52224)
0925 15:03:59.283101 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.283205 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:03:59.283699 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32827: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
W20260925 15:03:59.414994 27922 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:03:59.415349 27922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:59.415460 27922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:59.415593 27922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:59.422472 27922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:59.422544 27922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:59.422662 27922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:59.422734 27922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:59.422818 27922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:59.426957 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } attempt: 4
I20260925 15:03:59.428175 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.427569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52230)
0925 15:03:59.427898 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.428004 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 15:03:59.429210 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.428835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52234)
0925 15:03:59.428985 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.429066 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:03:59.429388 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.253:32827: Network error: Client connection negotiation failed: client connection to 127.4.243.253:32827: connect: Connection refused (error 111)
I20260925 15:03:59.433050 27922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35159,127.4.243.253:32827,127.4.243.252:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.253.keytab
--principal=kudu/127.4.243.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.253:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.243.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:59.434351 27922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:59.435657 27922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:59.446794 27930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:59.446942 27927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:59.446875 27928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:59.448058 27922 server_base.cc:1034] running on GCE node
I20260925 15:03:59.448766 27922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:59.450045 27922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:59.451280 27922 hybrid_clock.cc:648] HybridClock initialized: now 1790348639451218 us; error 39 us; skew 500 ppm
I20260925 15:03:59.451591 27922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:59 dist-test-slave-qxh0 krb5kdc[27835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:59.454582 27922 init.cc:377] Logged in from keytab as kudu/127.4.243.253@KRBTEST.COM (short username kudu)
I20260925 15:03:59.455914 27922 webserver.cc:460] Webserver started at http://127.4.243.253:34663/ using document root <none> and password file <none>
I20260925 15:03:59.456563 27922 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:59.456688 27922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:59.456991 27922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:59.459843 27922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data/instance:
uuid: "b6899571912c4fcd9d8a3d27029a0d7a"
format_stamp: "Formatted at 2026-09-25 15:03:59 on dist-test-slave-qxh0"
server_key: "5d2d949575e456df1ce768f4a4ad0c8e"
server_key_iv: "3e6e28c928d070693ecc36b9047cae0b"
server_key_version: "encryptionkey@0"
I20260925 15:03:59.460606 27922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/wal/instance:
uuid: "b6899571912c4fcd9d8a3d27029a0d7a"
format_stamp: "Formatted at 2026-09-25 15:03:59 on dist-test-slave-qxh0"
server_key: "5d2d949575e456df1ce768f4a4ad0c8e"
server_key_iv: "3e6e28c928d070693ecc36b9047cae0b"
server_key_version: "encryptionkey@0"
I20260925 15:03:59.465366 27922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:03:59.468768 27937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:59.470065 27922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:03:59.470265 27922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/wal
uuid: "b6899571912c4fcd9d8a3d27029a0d7a"
format_stamp: "Formatted at 2026-09-25 15:03:59 on dist-test-slave-qxh0"
server_key: "5d2d949575e456df1ce768f4a4ad0c8e"
server_key_iv: "3e6e28c928d070693ecc36b9047cae0b"
server_key_version: "encryptionkey@0"
I20260925 15:03:59.470476 27922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:59.486989 27922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:59.491053 27922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:59.491313 27922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:59.508172 27922 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.253:32827
I20260925 15:03:59.508190 27988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.253:32827 every 8 connection(s)
I20260925 15:03:59.509411 27922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/data/info.pb
I20260925 15:03:59.512918 27989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:03:59.514832 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27922
I20260925 15:03:59.514994 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-1/wal/instance
I20260925 15:03:59.515452 5071 external_mini_cluster.cc:1428] Setting key 7707bebf5fce7cf536cd42de8e8726a4
I20260925 15:03:59.519320 27989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:03:59 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.243.252@KRBTEST.COM; defaulting to no policy
I20260925 15:03:59.541980 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.521900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.4.243.254:35159)
0925 15:03:59.522073 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:59.522078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:59.523301 (+ 1223us) server_negotiation.cc:407] Connection header received
0925 15:03:59.526240 (+ 2939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:59.526246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:59.526317 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:59.526410 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:59.528409 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.529073 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.531456 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.531876 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.531970 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.535958 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:59.535968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:59.535976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:59.536022 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:59.538813 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.539513 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.539518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.539520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.539634 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.540024 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.540029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.540036 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.540566 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:59.540718 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:59.541408 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 15:03:59.541628 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.243.252@KRBTEST.COM" created.
I20260925 15:03:59.542997 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.522348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35159 (local address 127.0.0.1:55832)
0925 15:03:59.523090 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.523161 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:59.525912 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:59.526697 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:59.526710 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:59.527479 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:59.528223 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.528250 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.529234 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.529243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.531312 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.531328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.532091 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.532100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.532413 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.533130 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:59.533164 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:59.535718 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:59.538961 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.538969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.538983 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.539387 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.539745 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.539749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.539753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.539911 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.540712 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:59.540720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:59.541011 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:03:59.541297 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 15:03:59.541749 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":329,"threads_started":1}
I20260925 15:03:59.544684 27989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:03:59 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:03:59.558858 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.545434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52248)
0925 15:03:59.545632 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.545653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:59.545754 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:59.546516 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:59.546523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:59.547021 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:59.547214 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.547227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.548506 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.548513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.549912 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.549928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.550785 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.550791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.550966 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.551519 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:59.551535 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:59.553410 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:59.556490 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.556497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.556501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.556836 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.557191 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.557197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.557201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.557334 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.557895 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:59.557903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:59.558032 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 15:03:59.558176 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 15:03:59.558443 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:03:59.559154 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.545527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.4.243.253:32827)
0925 15:03:59.546124 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:59.546130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:59.546154 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:03:59.546198 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:59.546203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:59.546358 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:59.546466 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:59.547400 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.548375 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.550062 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.550579 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.550768 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.553603 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:59.553613 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:59.553626 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:59.553655 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:59.556351 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.556967 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.556979 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.556981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.557085 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.557432 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.557439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.557442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.557776 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:59.557980 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:59.558338 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 15:03:59.558538 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":314,"threads_started":1}
I20260925 15:03:59.561514 27989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:59.562716 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.562259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50336)
0925 15:03:59.562435 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.562533 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:03:59.562968 27940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.564405 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.564108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50340)
0925 15:03:59.564239 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.564317 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:03:59.564625 27989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.243.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab.
Entry for principal kudu/127.4.243.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab.
I20260925 15:03:59.571926 5071 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
/tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:36051
--webserver_interface=127.4.243.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab
--principal=kudu/127.4.243.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.243.212:43539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.243.254:35159,127.4.243.253:32827,127.4.243.252:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:03:59.613885 27989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 1
I20260925 15:03:59.615085 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.614651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50348)
0925 15:03:59.614835 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.614944 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:03:59.616078 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.615755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50352)
0925 15:03:59.615909 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.615978 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:03:59.616286 27989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.684484 27989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 2
I20260925 15:03:59.685562 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.685085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50358)
0925 15:03:59.685316 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.685414 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:03:59.686491 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.686176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50372)
0925 15:03:59.686315 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.686383 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:03:59.686717 27989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.728655 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } attempt: 5
Sep 25 15:03:59 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:03:59.743141 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.729395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52262)
0925 15:03:59.729612 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.729641 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 15:03:59.729778 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:03:59.730098 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:03:59.730104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:03:59.730625 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:03:59.730898 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.730915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.731886 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.731892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.733348 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:03:59.733367 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.734111 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.734118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:03:59.734388 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.735117 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:03:59.735142 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:03:59.737809 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:03:59.740675 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.740680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.740684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.741139 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.741468 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:03:59.741473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:03:59.741478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:03:59.741666 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:03:59.742139 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:03:59.742144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:03:59.742324 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 15:03:59.742507 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 15:03:59.742788 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:03:59.743151 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.729628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.4.243.253:32827)
0925 15:03:59.729795 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:03:59.729799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:03:59.729817 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:03:59.729889 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:03:59.729892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:03:59.729963 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:03:59.730038 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:03:59.731052 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.731760 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.733517 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:03:59.733940 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:03:59.734029 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:03:59.737978 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:03:59.737982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:03:59.737985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:03:59.738016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:03:59.740539 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.741242 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.741246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.741248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.741359 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:03:59.741769 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:03:59.741772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:03:59.741774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:03:59.742024 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:03:59.742124 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:03:59.742642 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 15:03:59.742840 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:03:59.744357 27915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:03:59.745541 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.745028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50382)
0925 15:03:59.745222 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.745345 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:03:59.746505 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.746210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50396)
0925 15:03:59.746339 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.746410 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:03:59.746762 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.777930 27989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 3
I20260925 15:03:59.779038 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.778567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50404)
0925 15:03:59.778804 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.778893 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 15:03:59.780050 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.779637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50420)
0925 15:03:59.779884 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.779948 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260925 15:03:59.780242 27989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.797968 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 1
I20260925 15:03:59.799034 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.798627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50424)
0925 15:03:59.798804 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.798893 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:03:59.799916 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.799627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50432)
0925 15:03:59.799756 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.799820 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:03:59.800103 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.868294 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 2
I20260925 15:03:59.869402 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.868903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50448)
0925 15:03:59.869133 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.869246 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:03:59.870322 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.870005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50464)
0925 15:03:59.870140 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.870224 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:03:59.870501 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
W20260925 15:03:59.886705 27996 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:03:59.887024 27996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:03:59.887146 27996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:03:59.887286 27996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:03:59.894030 27996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:03:59.894143 27996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:03:59.894258 27996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:03:59.894336 27996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:03:59.894433 27996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:03:59.903681 27996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.243.212:43539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.243.254:35159,127.4.243.253:32827,127.4.243.252:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/krb5kdc/kudu_127.4.243.252.keytab
--principal=kudu/127.4.243.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.243.252:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.243.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 13:59:48 UTC on ccycloud.quasar-hqgcrl.root.comops.site
build id 4672724
ASAN enabled
I20260925 15:03:59.905062 27996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:03:59.906390 27996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:03:59.918228 28001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:03:59.918231 28002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:59.919792 27996 server_base.cc:1034] running on GCE node
W20260925 15:03:59.918228 28004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:03:59.922721 27996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:03:59.923416 27989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 4
I20260925 15:03:59.924815 27996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:03:59.924845 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.924049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50472)
0925 15:03:59.924396 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.924544 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:03:59.926076 27996 hybrid_clock.cc:648] HybridClock initialized: now 1790348639925954 us; error 94 us; skew 500 ppm
I20260925 15:03:59.926244 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.925864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50486)
0925 15:03:59.926002 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.926090 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:03:59.926470 27989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.926525 27996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:03:59 dist-test-slave-qxh0 krb5kdc[27835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:03:59.929735 27996 init.cc:377] Logged in from keytab as kudu/127.4.243.252@KRBTEST.COM (short username kudu)
I20260925 15:03:59.931195 27996 webserver.cc:460] Webserver started at http://127.4.243.252:34337/ using document root <none> and password file <none>
I20260925 15:03:59.931835 27996 fs_manager.cc:359] Metadata directory not provided
I20260925 15:03:59.931941 27996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:03:59.932293 27996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:03:59.935221 27996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data/instance:
uuid: "076588e6450846d9b895d39c1ab299b5"
format_stamp: "Formatted at 2026-09-25 15:03:59 on dist-test-slave-qxh0"
server_key: "5cebeed3c2825030d578d472e8bfb567"
server_key_iv: "7b5abe96f8ab7beb2e3ea95b13ad1a59"
server_key_version: "encryptionkey@0"
I20260925 15:03:59.936093 27996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/wal/instance:
uuid: "076588e6450846d9b895d39c1ab299b5"
format_stamp: "Formatted at 2026-09-25 15:03:59 on dist-test-slave-qxh0"
server_key: "5cebeed3c2825030d578d472e8bfb567"
server_key_iv: "7b5abe96f8ab7beb2e3ea95b13ad1a59"
server_key_version: "encryptionkey@0"
I20260925 15:03:59.941120 27996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:03:59.944860 28011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:03:59.946126 27996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:03:59.946312 27996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data,/tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/wal
uuid: "076588e6450846d9b895d39c1ab299b5"
format_stamp: "Formatted at 2026-09-25 15:03:59 on dist-test-slave-qxh0"
server_key: "5cebeed3c2825030d578d472e8bfb567"
server_key_iv: "7b5abe96f8ab7beb2e3ea95b13ad1a59"
server_key_version: "encryptionkey@0"
I20260925 15:03:59.946527 27996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:03:59.960094 27996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:03:59.963905 27996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:03:59.964273 27996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:03:59.976744 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 3
I20260925 15:03:59.982739 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.982214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50488)
0925 15:03:59.982393 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.982503 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:03:59.984050 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:03:59.983588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50502)
0925 15:03:59.983844 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 15:03:59.983916 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260925 15:03:59.984308 27915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.243.252:36051: Network error: Client connection negotiation failed: client connection to 127.4.243.252:36051: connect: Connection refused (error 111)
I20260925 15:03:59.992138 27996 rpc_server.cc:307] RPC server started. Bound to: 127.4.243.252:36051
I20260925 15:03:59.992187 28062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.243.252:36051 every 8 connection(s)
I20260925 15:03:59.993542 27996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/data/info.pb
I20260925 15:03:59.998248 28063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:04:00.003598 5071 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu as pid 27996
I20260925 15:04:00.003803 5071 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Tl3Wl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284481048-5071-0/minicluster-data/master-2/wal/instance
I20260925 15:04:00.004208 5071 external_mini_cluster.cc:1428] Setting key 76c1c4f9e8a87a1aff52fe58c2959f4d
I20260925 15:04:00.008342 28063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
I20260925 15:04:00.028890 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.005939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.4.243.252:36051)
0925 15:04:00.006756 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.006767 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.006808 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:04:00.008530 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.008552 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.008932 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.009313 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.010812 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.013031 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.015030 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.015853 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.016029 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.020468 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.020477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.020488 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.020521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.023356 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.024204 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.024212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.024220 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.024402 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.024860 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.024866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.024871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.025484 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.025746 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.026429 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.026863 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":355,"threads_started":1}
I20260925 15:04:00.039389 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.011555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.4.243.254:35159)
0925 15:04:00.011704 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.011709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.013890 (+ 2181us) server_negotiation.cc:407] Connection header received
0925 15:04:00.014141 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.014146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.014214 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.014312 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.015531 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.016734 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.018722 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.019536 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.019679 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.031816 (+ 12137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.031825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.031832 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.031876 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.035799 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.036925 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.036931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.036935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.037124 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.037572 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.037577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.037580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.037950 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.038099 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.038716 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.038929 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:04:00.040524 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.011482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.254:35159 (local address 127.0.0.1:55836)
0925 15:04:00.013764 (+ 2282us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:00.013808 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:00.013995 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:00.014363 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:00.014370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:00.015030 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:00.015364 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.015375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.016903 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.016908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.018539 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.018557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.020003 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.020012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.020246 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.026784 (+ 6538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:00.026804 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:00.031573 (+ 4769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:00.035942 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.035961 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.035972 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.036761 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.037267 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.037271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.037275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.037436 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.038188 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:00.038195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:00.038374 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:04:00.039645 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0925 15:04:00.040026 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2068,"thread_start_us":1931,"threads_started":1}
I20260925 15:04:00.042789 28063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.254@KRBTEST.COM
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
I20260925 15:04:00.055768 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.033070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.4.243.254:35159)
0925 15:04:00.038136 (+ 5066us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.038140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.038161 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:04:00.038198 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.038202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.038253 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.038375 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.039713 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.040908 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.043207 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.043835 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.043975 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.049576 (+ 5601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.049582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.049587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.049628 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.052826 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.053418 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.053423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.053426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.053558 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.054024 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.054029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.054033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.054389 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.054689 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.055084 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.055328 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4951,"thread_start_us":4779,"threads_started":1}
I20260925 15:04:00.059068 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.043639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.4.243.253:32827)
0925 15:04:00.043814 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.043819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.043837 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:04:00.043981 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.043987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.044067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.044167 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.044935 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.045611 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.047557 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.047991 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.048123 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.051730 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.051737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.051743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.051790 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.055355 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.056584 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.056592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.056598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.056798 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.057198 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.057203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.057207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.057530 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.057709 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.058314 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.058561 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:04:00.059096 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.043530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.253:32827 (local address 127.0.0.1:52266)
0925 15:04:00.043715 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:00.043735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:00.043833 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:00.044205 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:00.044210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:00.044647 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:00.044804 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.044814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.045750 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.045757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.047378 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.047395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.048135 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.048141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.048347 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.048886 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:00.048902 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:00.051272 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:00.055507 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.055514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.055518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.056427 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.056907 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.056911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.056914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.057092 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.057786 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:00.057794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:00.057991 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 15:04:00.058203 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 15:04:00.058575 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:04:00.060420 28063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.243.253@KRBTEST.COM
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.252@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:04:00.074939 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.061235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50508)
0925 15:04:00.061429 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:00.061444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:00.061539 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:00.061929 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:00.061934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:00.062374 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:00.062520 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.062530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.063928 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.063934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.064731 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.064741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.066025 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.066032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.066309 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.066937 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:00.066962 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:00.069434 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:00.072640 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.072647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.072653 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.072986 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.073310 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.073316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.073322 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.073441 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.073920 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:00.073924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:00.074052 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 15:04:00.074225 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 15:04:00.074514 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260925 15:04:00.075088 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.057443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.4.243.253:32827)
0925 15:04:00.059758 (+ 2315us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.059763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.059780 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:04:00.059831 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.059835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.059902 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.060005 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.061771 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.062857 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.063984 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.064702 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.064826 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.069638 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.069643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.069646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.069676 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.072406 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.073035 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.073042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.073045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.073181 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.073539 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.073545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.073548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.073846 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.074091 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.074474 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.074714 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2196,"thread_start_us":2116,"threads_started":1}
I20260925 15:04:00.075474 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.061236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.4.243.252:36051)
0925 15:04:00.061394 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.061399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.061505 (+ 106us) server_negotiation.cc:407] Connection header received
0925 15:04:00.061622 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.061626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.061684 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.061773 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.062704 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.063794 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.064890 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.065862 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.065991 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.069665 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.069672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.069677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.069715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.072502 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.073093 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.073100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.073103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.073210 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.073534 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.073541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.073544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.073799 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.073940 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.074319 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.074720 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":60}
I20260925 15:04:00.079648 5071 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:04:00.080816 28063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: Bootstrap starting.
I20260925 15:04:00.084798 28063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:00.081943 27890 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "db5d2a45e55642cfa4dc1cfddff490e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55848
I20260925 15:04:00.086040 28063 log.cc:824] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:00.088902 28063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: No bootstrap required, opened a new log
I20260925 15:04:00.099059 28063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.099414 28063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076588e6450846d9b895d39c1ab299b5, State: Initialized, Role: FOLLOWER
I20260925 15:04:00.099900 28063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.100785 28071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.101101 28071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:00.101861 28063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:04:00.110831 28082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:04:00.110957 28082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:04:00.108598 27996 master_runner.cc:186] Error getting master registration for 127.4.243.254:35159: OK, instance_id { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" instance_seqno: 1790348639018257 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:04:00.119699 27996 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:04:00.161535 27915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 4
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.254@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:04:00.177799 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.162293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50514)
0925 15:04:00.162528 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:00.162552 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:00.162727 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:00.163093 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:00.163099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:00.163635 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:00.163855 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.163867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.165354 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.165359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.166919 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.166934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.167925 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.167931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.168137 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.168833 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:00.168857 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:00.171485 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:00.175395 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.175405 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.175407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.175783 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.176124 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.176131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.176136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.176284 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.176859 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:00.176863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:00.177026 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 15:04:00.177161 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:04:00.177422 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:04:00.177812 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.162352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.4.243.252:36051)
0925 15:04:00.162652 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.162661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.162700 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:04:00.162853 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.162858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.162935 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.163048 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.164006 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.165186 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.167055 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.167774 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.167893 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.171673 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.171679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.171684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.171724 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.175230 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.175899 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.175904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.175908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.176021 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.176392 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.176397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.176404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.176744 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.176909 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.177260 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.177455 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 15:04:00.183629 27915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4: Bootstrap starting.
I20260925 15:04:00.187696 27915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:00.188738 27915 log.cc:824] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:00.191464 27915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4: No bootstrap required, opened a new log
I20260925 15:04:00.196444 27915 raft_consensus.cc:348] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.196795 27915 raft_consensus.cc:729] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5d2a45e55642cfa4dc1cfddff490e4, State: Initialized, Role: FOLLOWER
I20260925 15:04:00.197213 27915 consensus_queue.cc:260] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.198093 28085 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.198446 28085 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:00.199222 27915 sys_catalog.cc:564] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:00.199388 27890 raft_consensus.cc:482] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:04:00.199627 27890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:00.203249 27890 raft_consensus.cc:504] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.206794 28038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "076588e6450846d9b895d39c1ab299b5"
I20260925 15:04:00.207247 28038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:00.210505 28038 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5d2a45e55642cfa4dc1cfddff490e4 in term 1.
I20260925 15:04:00.211436 27866 leader_election.cc:302] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 076588e6450846d9b895d39c1ab299b5, db5d2a45e55642cfa4dc1cfddff490e4; no voters:
I20260925 15:04:00.211875 28085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [term 1 FOLLOWER]: Leader election won for term 1
W20260925 15:04:00.216177 28017 tablet.cc:2369] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:00.218037 27890 leader_election.cc:288] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [CANDIDATE]: Term 1 election: Requested vote from peers b6899571912c4fcd9d8a3d27029a0d7a (127.4.243.253:32827), 076588e6450846d9b895d39c1ab299b5 (127.4.243.252:36051)
I20260925 15:04:00.222136 28085 raft_consensus.cc:686] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [term 1 LEADER]: Becoming Leader. State: Replica: db5d2a45e55642cfa4dc1cfddff490e4, State: Running, Role: LEADER
I20260925 15:04:00.222819 28085 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.225678 27989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } attempt: 5
W20260925 15:04:00.226276 28096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:04:00.226505 28096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 15:04:00 dist-test-slave-qxh0 krb5kdc[27835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.243.253@KRBTEST.COM for kudu/127.4.243.252@KRBTEST.COM
I20260925 15:04:00.240154 28098 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5d2a45e55642cfa4dc1cfddff490e4. Latest consensus state: current_term: 1 leader_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.240666 28098 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:00.251260 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.226262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.243.252:36051 (local address 127.0.0.1:50528)
0925 15:04:00.226660 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 15:04:00.226681 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:04:00.226797 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:04:00.227376 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:04:00.227382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:04:00.227986 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:04:00.228254 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.228271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.229985 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.229993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.231755 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:04:00.231775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.233190 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.233198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:04:00.233494 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.234246 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:04:00.234277 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:04:00.237641 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:04:00.244788 (+ 7147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.244793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.244797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.245568 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.246260 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:04:00.246265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:04:00.246269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:04:00.246518 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:04:00.247731 (+ 1213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:04:00.247737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:04:00.247988 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 15:04:00.248185 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 15:04:00.248608 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260925 15:04:00.254400 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.226634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.4.243.252:36051)
0925 15:04:00.226831 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.226840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.226868 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:04:00.226944 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.226952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.227040 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.227147 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.228469 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.229808 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.231952 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.232815 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.232947 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.237896 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.237905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.237928 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.237988 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.244480 (+ 6492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.245783 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.245819 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.245823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.246048 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.246782 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.246789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.246797 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.247359 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.249166 (+ 1807us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.250256 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.252373 (+ 2117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 15:04:00.258220 27869 tablet.cc:2369] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:04:00.213228 27848 master_runner.cc:186] Error getting master registration for 127.4.243.253:32827: OK, instance_id { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" instance_seqno: 1790348639503113 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:04:00.260741 27989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a: Bootstrap starting.
I20260925 15:04:00.260996 27848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:04:00.264753 27989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a: Neither blocks nor log segments found. Creating new log.
I20260925 15:04:00.266034 27989 log.cc:824] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:04:00.270287 28103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:04:00.274070 28103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:04:00.289940 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.260939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.4.243.254:35159)
0925 15:04:00.261196 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.261216 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.271804 (+ 10588us) server_negotiation.cc:407] Connection header received
0925 15:04:00.272013 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.272019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.272096 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.272185 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.274286 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.275188 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.277021 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.277818 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.277947 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.281867 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.281875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.281882 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.281924 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.286302 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.287246 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.287255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.287261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.287400 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.287838 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.287842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.287846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.288209 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.288472 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.288956 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.289328 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:04:00.304253 28038 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [term 1 FOLLOWER]: Refusing update from remote peer db5d2a45e55642cfa4dc1cfddff490e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:00.305259 28098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:00.310050 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.268326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.4.243.253:32827)
0925 15:04:00.268673 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.268681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.289510 (+ 20829us) server_negotiation.cc:407] Connection header received
0925 15:04:00.289760 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.289766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.289847 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.289920 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.292482 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.293160 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.295091 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.295788 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.295896 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.299390 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.299396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.299400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.299437 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.302818 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.304448 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.304456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.304490 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.304908 (+ 418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.307908 (+ 3000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.307915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.307920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.308369 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.308755 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.309354 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.309613 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260925 15:04:00.310379 27989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a: No bootstrap required, opened a new log
I20260925 15:04:00.315949 28071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5d2a45e55642cfa4dc1cfddff490e4. Latest consensus state: current_term: 1 leader_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.316264 28071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:00.319619 28109 mvcc.cc:205] Tried to move back new op lower bound from 7333268030666047488 to 7333268030365323264. Current Snapshot: MvccSnapshot[applied={T|T < 7333268030666047488}]
I20260925 15:04:00.322553 28085 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.322893 28085 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:00.325615 27989 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.326007 27989 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6899571912c4fcd9d8a3d27029a0d7a, State: Initialized, Role: FOLLOWER
I20260925 15:04:00.326526 27989 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } }
I20260925 15:04:00.326936 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.268488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.4.243.252:36051)
0925 15:04:00.270397 (+ 1909us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.270404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.297048 (+ 26644us) server_negotiation.cc:407] Connection header received
0925 15:04:00.297314 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.297320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.297413 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.297480 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.299312 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.300283 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.302538 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.303248 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.303362 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.313832 (+ 10470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:04:00.313841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:04:00.313847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:04:00.313892 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:04:00.320238 (+ 6346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.321289 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.321297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.321303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.321461 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:04:00.322329 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:04:00.322333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:04:00.322336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:04:00.322705 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:04:00.322951 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.324550 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.325105 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1775}
I20260925 15:04:00.327520 28103 catalog_manager.cc:1347] Generated new cluster ID: 923bedbc97c44b298b11fddc0f5fce77
I20260925 15:04:00.327607 28103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:04:00.328467 28110 mvcc.cc:205] Tried to move back new op lower bound from 7333268030666047488 to 7333268030365323264. Current Snapshot: MvccSnapshot[applied={T|T < 7333268030666047488}]
I20260925 15:04:00.330856 28071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.331090 28071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:00.363152 28114 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.363685 28114 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:00.372287 27989 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:04:00.372999 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b6899571912c4fcd9d8a3d27029a0d7a"
I20260925 15:04:00.373934 27963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:04:00.377664 27963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [term 1 FOLLOWER]: Refusing update from remote peer db5d2a45e55642cfa4dc1cfddff490e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:04:00.378767 28098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:04:00.381179 27964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5d2a45e55642cfa4dc1cfddff490e4 in term 1.
I20260925 15:04:00.391845 28119 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.392359 28119 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:00.392231 28114 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5d2a45e55642cfa4dc1cfddff490e4. Latest consensus state: current_term: 1 leader_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.392508 28114 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:04:00.405622 28098 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d2a45e55642cfa4dc1cfddff490e4" member_type: VOTER last_known_addr { host: "127.4.243.254" port: 35159 } } peers { permanent_uuid: "b6899571912c4fcd9d8a3d27029a0d7a" member_type: VOTER last_known_addr { host: "127.4.243.253" port: 32827 } } peers { permanent_uuid: "076588e6450846d9b895d39c1ab299b5" member_type: VOTER last_known_addr { host: "127.4.243.252" port: 36051 } } }
I20260925 15:04:00.405992 28098 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:04:00.417974 28103 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:04:00.420004 28103 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:04:00.446687 28103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db5d2a45e55642cfa4dc1cfddff490e4: Generated new TSK 0
I20260925 15:04:00.447829 28103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:04:00.469875 28129 catalog_manager.cc:1259] Loaded cluster ID: 923bedbc97c44b298b11fddc0f5fce77
I20260925 15:04:00.469995 28129 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a: loading cluster ID for follower catalog manager: success
I20260925 15:04:00.474149 28129 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a: acquiring CA information for follower catalog manager: success
I20260925 15:04:00.476800 28129 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6899571912c4fcd9d8a3d27029a0d7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:04:00.606748 28068 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.568125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.4.243.254:35159)
0925 15:04:00.568336 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.568347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.603185 (+ 34838us) server_negotiation.cc:407] Connection header received
0925 15:04:00.605075 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.605082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.605176 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.605253 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.606444 (+ 1191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55868: BlockingRecv error: recv got EOF from 127.0.0.1:55868 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 15:04:00.609465 27993 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.574745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.4.243.253:32827)
0925 15:04:00.574965 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.574971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.607648 (+ 32677us) server_negotiation.cc:407] Connection header received
0925 15:04:00.608025 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.608031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.608110 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.608177 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.609234 (+ 1057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52304: BlockingRecv error: recv got EOF from 127.0.0.1:52304 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:04:00.611455 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.585073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.4.243.252:36051)
0925 15:04:00.585271 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.585279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.610049 (+ 24770us) server_negotiation.cc:407] Connection header received
0925 15:04:00.610194 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.610200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.610283 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.610380 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.611211 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50536: BlockingRecv error: recv got EOF from 127.0.0.1:50536 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:04:00.657007 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.637987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.4.243.253:32827)
0925 15:04:00.638164 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.638183 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.648731 (+ 10548us) server_negotiation.cc:407] Connection header received
0925 15:04:00.649240 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.649246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.649488 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.649620 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:00.650395 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.651605 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.653750 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.654481 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.654578 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.655533 (+ 955us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:00.656227 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:00.656342 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.656585 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.656683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:04:00.714164 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.700863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.4.243.252:36051)
0925 15:04:00.701066 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.701072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.712082 (+ 11010us) server_negotiation.cc:407] Connection header received
0925 15:04:00.712610 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.712617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.712774 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.712855 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.713898 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50544: BlockingRecv error: recv got EOF from 127.0.0.1:50544 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:04:00.743963 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.737252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.4.243.253:32827)
0925 15:04:00.737460 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.737466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.737968 (+ 502us) server_negotiation.cc:407] Connection header received
0925 15:04:00.738158 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.738169 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.738285 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.738390 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:00.738887 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.740008 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.741928 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.742631 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.742753 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.743174 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:00.743322 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:00.743418 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.743577 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.743657 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:04:00.760608 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.755220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.4.243.252:36051)
0925 15:04:00.755395 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.755401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.759156 (+ 3755us) server_negotiation.cc:407] Connection header received
0925 15:04:00.759355 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.759361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.759456 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.759566 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.760356 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50558: BlockingRecv error: recv got EOF from 127.0.0.1:50558 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:04:00.776916 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.765251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.4.243.253:32827)
0925 15:04:00.765432 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.765438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.771128 (+ 5690us) server_negotiation.cc:407] Connection header received
0925 15:04:00.771304 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.771310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.771392 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.771484 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:00.771926 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.772959 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.774880 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.775597 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.775728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.776136 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:00.776283 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:00.776377 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.776523 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.776599 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:04:00.790354 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.788226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.4.243.252:36051)
0925 15:04:00.788400 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.788405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.788916 (+ 511us) server_negotiation.cc:407] Connection header received
0925 15:04:00.789083 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.789090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.789179 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.789282 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.790121 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50560: BlockingRecv error: recv got EOF from 127.0.0.1:50560 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:04:00.805331 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.798126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.4.243.253:32827)
0925 15:04:00.798381 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.798387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.798913 (+ 526us) server_negotiation.cc:407] Connection header received
0925 15:04:00.799103 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.799109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.799215 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.799319 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:00.799928 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.801085 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.803092 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.803796 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.803897 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.804519 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:00.804649 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:00.804708 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.804943 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.805039 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260925 15:04:00.815044 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.812741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.4.243.252:36051)
0925 15:04:00.813022 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.813028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.813570 (+ 542us) server_negotiation.cc:407] Connection header received
0925 15:04:00.813756 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.813762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.813867 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.813991 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.814812 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50574: BlockingRecv error: recv got EOF from 127.0.0.1:50574 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260925 15:04:00.832540 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.825981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.4.243.253:32827)
0925 15:04:00.826157 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.826163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.826662 (+ 499us) server_negotiation.cc:407] Connection header received
0925 15:04:00.826833 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.826839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.826928 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.827036 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:00.827474 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.828565 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.830412 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.831096 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.831227 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.831734 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:00.831876 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:00.831972 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.832141 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.832231 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 15:04:00.842526 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.840169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.4.243.252:36051)
0925 15:04:00.840356 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.840362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.840878 (+ 516us) server_negotiation.cc:407] Connection header received
0925 15:04:00.841153 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.841160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.841249 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.841319 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.842318 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50580: BlockingRecv error: recv got EOF from 127.0.0.1:50580 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:04:00.869622 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.862543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.4.243.253:32827)
0925 15:04:00.862761 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.862766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.863324 (+ 558us) server_negotiation.cc:407] Connection header received
0925 15:04:00.863588 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.863593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.863684 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.863794 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:00.864292 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.865477 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.867525 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.868189 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.868322 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.868812 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:00.868971 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:00.869071 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.869246 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.869333 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260925 15:04:00.879226 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.876704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.4.243.252:36051)
0925 15:04:00.876919 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.876926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.877481 (+ 555us) server_negotiation.cc:407] Connection header received
0925 15:04:00.877772 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.877780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.877880 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.877955 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.878981 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50596: BlockingRecv error: recv got EOF from 127.0.0.1:50596 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:04:00.923079 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:00.916226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.4.243.253:32827)
0925 15:04:00.916424 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.916431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.916986 (+ 555us) server_negotiation.cc:407] Connection header received
0925 15:04:00.917248 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.917255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.917354 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.917464 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:00.917907 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.918987 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.920981 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:00.921670 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:00.921796 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:00.922297 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:00.922411 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:00.922495 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:00.922669 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 15:04:00.922751 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 15:04:00.932371 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:00.929981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.4.243.252:36051)
0925 15:04:00.930150 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:00.930156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:00.930955 (+ 799us) server_negotiation.cc:407] Connection header received
0925 15:04:00.931099 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:00.931104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:00.931193 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:00.931290 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:00.932146 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50602: BlockingRecv error: recv got EOF from 127.0.0.1:50602 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:04:01.006161 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:01.000078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.4.243.253:32827)
0925 15:04:01.000304 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:01.000310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:01.000643 (+ 333us) server_negotiation.cc:407] Connection header received
0925 15:04:01.000823 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:01.000828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:01.000905 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:01.000987 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:01.001499 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.002420 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.004390 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.004873 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.005027 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:01.005446 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:01.005545 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:01.005612 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:01.005771 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:04:01.005833 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 15:04:01.015383 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:04:01.013657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.4.243.252:36051)
0925 15:04:01.013845 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:01.013850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:01.014186 (+ 336us) server_negotiation.cc:407] Connection header received
0925 15:04:01.014350 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:01.014354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:01.014445 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:01.014521 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:04:01.015240 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50614: BlockingRecv error: recv got EOF from 127.0.0.1:50614 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:04:01.112887 28082 catalog_manager.cc:1259] Loaded cluster ID: 923bedbc97c44b298b11fddc0f5fce77
I20260925 15:04:01.112999 28082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: loading cluster ID for follower catalog manager: success
I20260925 15:04:01.115722 28082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: acquiring CA information for follower catalog manager: success
I20260925 15:04:01.117455 28082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 076588e6450846d9b895d39c1ab299b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:04:01.151773 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:01.146673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.4.243.253:32827)
0925 15:04:01.146911 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:01.146917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:01.147163 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:04:01.147416 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:01.147422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:01.147496 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:01.147587 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:01.147953 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.148659 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.150103 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.150535 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.150733 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:01.151086 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:01.151187 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:01.151278 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:01.151415 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:04:01.151493 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:04:01.162364 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:01.157203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.4.243.252:36051)
0925 15:04:01.157450 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:01.157456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:01.157689 (+ 233us) server_negotiation.cc:407] Connection header received
0925 15:04:01.157828 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:01.157831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:01.157990 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:01.158087 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:01.158443 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.159207 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.160421 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.160797 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.160884 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:01.161322 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:01.161797 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:01.161901 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:01.161993 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:04:01.162053 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:04:01.175581 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:01.168078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.4.243.253:32827)
0925 15:04:01.168226 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:01.168231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:01.169945 (+ 1714us) server_negotiation.cc:407] Connection header received
0925 15:04:01.170108 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:01.170113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:01.170225 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:01.170305 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:01.171192 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.171922 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.173617 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.174179 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.174304 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:01.174847 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:01.174980 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:01.175060 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:01.175203 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 15:04:01.175279 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:04:01.176152 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:01.168043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.4.243.252:36051)
0925 15:04:01.168222 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:01.168228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:01.169738 (+ 1510us) server_negotiation.cc:407] Connection header received
0925 15:04:01.170053 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:01.170058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:01.170147 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:01.170247 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:01.170908 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.171681 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.174002 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.174696 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.174832 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:01.175458 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:01.175545 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:01.175592 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:01.175829 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:04:01.175906 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:04:01.177768 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:04:01.167783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.4.243.254:35159)
0925 15:04:01.169994 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0925 15:04:01.170000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:04:01.170020 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:04:01.170139 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:04:01.170149 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:04:01.170364 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:04:01.170542 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:04:01.172183 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.173312 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.175510 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:04:01.176091 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:04:01.176221 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:04:01.176744 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:04:01.177132 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:04:01.177240 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:04:01.177388 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:04:01.177481 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":1976,"threads_started":1}
I20260925 15:04:01.182196 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27848
I20260925 15:04:01.183377 27910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:01.292225 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27848
I20260925 15:04:01.312925 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27922
I20260925 15:04:01.313992 27984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:01.420830 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27922
I20260925 15:04:01.441099 5071 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu pid 27996
I20260925 15:04:01.442173 28058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:04:01.548548 5071 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Tl3Wl/build/asan/bin/kudu with pid 27996
2026-09-25T15:04:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4280 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8513 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357030 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST